[ 5379.507175] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5386.220618] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5395.686281] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5397.706562] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5400.139181] Lustre: Failing over lustre-MDT0000
[ 5400.587204] Lustre: server umount lustre-MDT0000 complete
[ 5401.094650] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5401.111786] LustreError: 6531:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5401.143387] LustreError: 6531:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages
[ 5417.951590] Lustre: 3656:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846874/real 1787846874]  req@ffff9ffb11410000 x1874687811683072/t0(0) o400->MGC192.168.202.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787846890 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5418.005538] Lustre: 3656:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
[ 5418.014565] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5425.190867] LDISKFS-fs (dm-0): 7 truncates cleaned up
[ 5425.192913] LDISKFS-fs (dm-0): recovery complete
[ 5425.209901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5428.212301] LustreError: 3655:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9ffb37f3da40 x1874687811693824/t0(0) o250->MGC192.168.202.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5428.678659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5428.753819] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5430.972519] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5433.960972] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
[ 5434.011855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:679 to 0x280000401:705)
[ 5434.012794] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:611 to 0x2c0000401:641)
[ 5434.336160] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5445.135324] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5446.931268] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5453.724982] Lustre: DEBUG MARKER: (dd_pid=128338, time=0, timeout=600)
[ 5496.104287] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5502.577979] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5510.994509] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5512.826830] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5515.139542] Lustre: Failing over lustre-MDT0000
[ 5515.745912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5515.759395] Lustre: Skipped 4 previous similar messages
[ 5515.768183] LustreError: 6531:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5515.786716] LustreError: 6531:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 33 previous similar messages
[ 5515.870614] Lustre: server umount lustre-MDT0000 complete
[ 5536.230190] Lustre: 3658:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846993/real 1787846993]  req@ffff9ffb3b96a1c0 x1874687811759104/t0(0) o400->MGC192.168.202.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787847009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5536.273472] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5539.666494] LDISKFS-fs (dm-0): 3 truncates cleaned up
[ 5539.681649] LDISKFS-fs (dm-0): recovery complete
[ 5539.721436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5546.471996] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3d3dcc65062271e7
[ 5546.480444] Lustre: MGC192.168.202.105@tcp: Connection restored to 0@lo (at 0@lo)
[ 5546.486028] Lustre: Skipped 5 previous similar messages
[ 5546.821194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5546.870946] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5548.371512] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5551.868900] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5552.207298] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 5552.262606] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:611 to 0x2c0000401:673)
[ 5552.266452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:707 to 0x280000401:737)
[ 5562.499654] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5564.440514] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5570.489332] Lustre: DEBUG MARKER: (dd_pid=130855, time=0, timeout=600)
[ 5618.683534] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 12:11:29 (1787847089)
[ 5665.253670] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 12:12:16 (1787847136)
[ 5691.136515] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5703.964605] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5705.790412] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5707.542944] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5709.906621] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5713.706607] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5718.112931] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5721.741059] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5725.526823] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5728.982919] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5732.423488] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5735.942212] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5784.338666] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 12:14:16 (1787847256)
[ 5797.857661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5797.864906] Lustre: Skipped 2 previous similar messages
[ 5802.979128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5802.982407] Lustre: Skipped 4 previous similar messages
[ 5804.116114] Lustre: server umount lustre-MDT0000 complete
[ 5808.059298] LustreError: 8459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847281 with bad export cookie 4412907944159637991
[ 5808.075802] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5808.078072] LustreError: 8459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 5808.126758] LustreError: 16796:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5808.131539] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 5808.139336] LustreError: 16796:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 37 previous similar messages
[ 5808.166461] Lustre: Skipped 1 previous similar message
[ 5808.464514] Lustre: server umount lustre-MDT0001 complete
[ 5815.299261] Lustre: server umount lustre-OST0000 complete
[ 5820.642483] Lustre: server umount lustre-OST0001 complete
[ 5838.617959] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 5850.784666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5851.334670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5856.959166] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5865.425327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5865.985191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 5871.130474] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5873.732224] Lustre: 155125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5880.417795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5880.807657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5887.343476] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5890.035726] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 5895.956665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 5896.166939] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up)
[ 5896.185981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:769)
[ 5896.365574] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5901.801202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:676 to 0x2c0000401:705)
[ 5902.627195] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5910.224423] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5914.642259] Lustre: 157000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5937.632190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5937.641170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5937.642920] LustreError: Skipped 1 previous similar message
[ 5937.664576] Lustre: Skipped 4 previous similar messages
[ 5942.774113] LustreError: 153967:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5942.782435] LustreError: 153967:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages
[ 5943.635741] Lustre: server umount lustre-MDT0000 complete
[ 5947.054242] LustreError: 153951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847420 with bad export cookie 4412907944159645187
[ 5947.066693] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5947.480603] Lustre: server umount lustre-MDT0001 complete
[ 5961.932153] Lustre: server umount lustre-OST0000 complete
[ 5963.744152] Lustre: 3656:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787847420/real 1787847420]  req@ffff9ffa07256d00 x1874687812031360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787847436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5965.752530] Lustre: server umount lustre-OST0001 complete
[ 5981.264291] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 5991.834742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5992.519677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5996.952723] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6005.233340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6010.174666] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6012.952555] Lustre: 160738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6018.955472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6025.548228] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6030.697848] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 6033.501409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6033.705697] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6033.708455] Lustre: Skipped 2 previous similar messages
[ 6037.824792] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:676 to 0x2c0000401:737)
[ 6037.826274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:801)
[ 6039.974827] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6048.407523] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6051.976476] Lustre: 162609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6060.768996] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 12:18:52 (1787847532)
[ 6062.213375] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 6144
[ 6063.604923] Lustre: DEBUG MARKER: run for 4MB test file
[ 6075.783706] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 6082.680638] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 6084.239754] Lustre: DEBUG MARKER: Write half of file
[ 6085.984799] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6087.775853] Lustre: DEBUG MARKER: Step1: done
[ 6089.410799] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 6091.162779] Lustre: DEBUG MARKER: Step2: done
[ 6122.520831] Lustre: DEBUG MARKER: OST0_SIZE: 3605396 required: 61440
[ 6124.421591] Lustre: DEBUG MARKER: run for 40MB test file
[ 6138.859385] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 6146.311426] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 6147.672269] Lustre: DEBUG MARKER: Write half of file
[ 6150.952511] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6155.014732] Lustre: DEBUG MARKER: Step1: done
[ 6156.982419] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 6158.614271] Lustre: DEBUG MARKER: Step2: done
[ 6210.302269] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 12:21:22 (1787847682)
[ 6253.053500] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 12:22:04 (1787847724)
[ 6289.366876] Lustre: DEBUG MARKER: Write...
[ 6291.753769] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6343.035489] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 12:23:34 (1787847814)
[ 6350.175233] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 12:23:42 (1787847822)
[ 6360.480940] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 12:23:51 (1787847831)
[ 6370.279950] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 12:24:01 (1787847841)
[ 6378.941883] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 12:24:10 (1787847850)
[ 6433.822503] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6580.499572] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6740.280541] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 12:30:11 (1787848211)
[ 6790.397674] Lustre: DEBUG MARKER: Restart...
[ 6795.747739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6795.752666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6795.770524] Lustre: Skipped 15 previous similar messages
[ 6800.354117] LustreError: 3656:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9ffb3f77ad00 x1874687812514304/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 6800.402991] LustreError: 3656:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff9ffa094f1e00 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2600 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 6800.869056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6800.885370] Lustre: Skipped 6 previous similar messages
[ 6801.898655] Lustre: server umount lustre-MDT0000 complete
[ 6806.008237] LustreError: 180623:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6806.017184] LustreError: 180623:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages
[ 6806.047528] LustreError: 159580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848279 with bad export cookie 4412907944159647490
[ 6806.059840] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6806.060603] LustreError: 159580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages
[ 6806.579476] Lustre: server umount lustre-MDT0001 complete
[ 6820.509561] Lustre: server umount lustre-OST0000 complete
[ 6834.343337] Lustre: server umount lustre-OST0001 complete
[ 6850.535436] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 6862.751299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6863.378243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6869.088476] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6878.765414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6879.372241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6884.184888] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6887.924510] Lustre: 186968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6895.817620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6896.117222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6904.111643] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6904.324440] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 6908.412954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:812 to 0x280000401:833)
[ 6916.240934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6922.260961] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:746 to 0x2c0000401:769)
[ 6925.481257] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6934.875161] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6939.872120] Lustre: 188839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7012.946281] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 12:34:44 (1787848484)
[ 7069.342845] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 12:35:40 (1787848540)
[ 8507.056780] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 12:59:38 (1787849978)
[ 8522.723431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 8522.734128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8522.746482] Lustre: Skipped 2 previous similar messages
[ 8522.755354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8524.769361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8524.780409] LustreError: 3658:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9ffa0a861e00 x1874687815891072/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 8524.838991] LustreError: 3658:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:usr lqe: ffff9ffa09458c00 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2544 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 8526.970570] Lustre: server umount lustre-MDT0000 complete
[ 8529.891902] LustreError: 187332:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8529.916865] LustreError: 187332:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages
[ 8531.727299] LustreError: 193169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850004 with bad export cookie 4412907944159656625
[ 8531.742568] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8531.745111] LustreError: 193169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 8532.242199] Lustre: server umount lustre-MDT0001 complete
[ 8548.074929] Lustre: server umount lustre-OST0000 complete
[ 8550.893454] Lustre: 3656:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850008/real 1787850008]  req@ffff9ffa0a860b40 x1874687815894272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8550.931568] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8550.952020] Lustre: Skipped 3 previous similar messages
[ 8552.416107] Lustre: 3659:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850009/real 1787850009]  req@ffff9ffb3d97f480 x1874687815894528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8553.534353] Lustre: server umount lustre-OST0001 complete
[ 8572.286880] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 8584.409995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8585.000804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8585.003988] Lustre: Skipped 1 previous similar message
[ 8590.110215] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8595.428740] LustreError: 196281:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8595.446895] LustreError: 196281:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages
[ 8599.353929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8599.936994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 8604.242083] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8607.404769] Lustre: 197435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8615.586342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8616.050218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8623.114508] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8624.246832] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 8630.769337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5836 to 0x280000401:5857)
[ 8633.886772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8634.243421] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8639.478296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5770 to 0x2c0000401:5793)
[ 8641.410916] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8650.810695] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8654.896408] Lustre: 199309:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8682.721505] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 13:02:34 (1787850154)
[ 8712.702881] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 13:03:04 (1787850184)
[ 8742.667405] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 13:03:34 (1787850214)
[ 8778.974101] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 13:04:10 (1787850250)
[ 8823.961801] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 13:04:55 (1787850295)
[ 8906.406966] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 13:06:17 (1787850377)
[ 8944.547940] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 13:06:55 (1787850415)
[ 9015.322372] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 13:08:06 (1787850486)
[ 9022.144629] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9022.147335] Lustre: Skipped 2 previous similar messages
[ 9024.161997] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9024.171992] Lustre: Skipped 91 previous similar messages
[ 9028.184441] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9028.190487] Lustre: Skipped 231 previous similar messages
[ 9036.193765] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9036.196085] Lustre: Skipped 429 previous similar messages
[ 9052.202724] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9052.208125] Lustre: Skipped 836 previous similar messages
[ 9084.223640] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9084.240304] Lustre: Skipped 1238 previous similar messages
[ 9148.281989] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9148.290187] Lustre: Skipped 2861 previous similar messages
[ 9445.737557] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9445.744129] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9445.746788] Lustre: Skipped 2322 previous similar messages
[ 9445.755956] Lustre: Skipped 1 previous similar message
[ 9550.822804] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9550.827324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9550.843582] Lustre: Skipped 3 previous similar messages
[ 9550.855433] Lustre: Skipped 5 previous similar messages
[ 9555.552462] Lustre: server umount lustre-MDT0000 complete
[ 9555.936621] LustreError: 196282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9555.965975] LustreError: 196282:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages
[ 9558.958665] LustreError: 196261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851032 with bad export cookie 4412907944161431223
[ 9558.971871] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9561.056866] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9561.061424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9561.070797] Lustre: Skipped 1 previous similar message
[ 9561.094854] Lustre: Skipped 2 previous similar messages
[ 9563.127800] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9568.230146] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 9568.246788] Lustre: Skipped 1 previous similar message
[ 9573.343356] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck?
[ 9573.750622] Lustre: server umount lustre-MDT0001 complete
[ 9578.541196] Lustre: server umount lustre-OST0000 complete
[ 9583.105493] Lustre: server umount lustre-OST0001 complete
[ 9590.314844] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid
[ 9598.462437] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 9651.201683] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[ 9662.226028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9662.525470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[ 9662.544408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 9662.622435] Lustre: lustre-MDT0000: new disk, initializing
[ 9662.747962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9662.763770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9667.059356] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9677.975743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 9678.102965] Lustre: 216132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17
[ 9678.136418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 9678.140390] Lustre: Skipped 1 previous similar message
[ 9678.218443] Lustre: lustre-MDT0001: new disk, initializing
[ 9678.282139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 9678.314206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 9678.324054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 9683.681740] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9688.458495] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[ 9695.182696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9695.483306] Lustre: lustre-OST0000: new disk, initializing
[ 9695.490213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 9695.501345] Lustre: 217767:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9695.655517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9697.058780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 9697.069991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 9697.120094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 9702.499347] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9714.730279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 9714.880680] Lustre: lustre-OST0001: new disk, initializing
[ 9714.885137] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 9714.890381] Lustre: 218637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0]
[ 9714.977659] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9716.513050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 9716.528870] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 9716.580228] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 9721.924736] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9733.368641] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9737.463526] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[ 9766.817827] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 13:20:38 (1787851238)
[ 9773.034204] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 13:20:44 (1787851244)
[ 9805.141875] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 13:21:16 (1787851276)
[ 9858.154830] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 13:22:09 (1787851329)
[ 9880.957332] Lustre: DEBUG MARKER: rename directory return 255
[ 9925.156735] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 13:23:16 (1787851396)
[ 9952.205681] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 13:23:43 (1787851423)
[ 9986.117438] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 13:24:17 (1787851457)
[10128.080065] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 13:26:39 (1787851599)
[10157.035874] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 13:27:08 (1787851628)
[10188.382906] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 13:27:39 (1787851659)
[10347.841665] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 13:30:19 (1787851819)
[10354.146076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[10354.168074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10354.190184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10354.196764] Lustre: Skipped 2 previous similar messages
[10356.707909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10356.712365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10356.725621] Lustre: Skipped 1 previous similar message
[10356.777960] Lustre: Skipped 1 previous similar message
[10359.313269] Lustre: server umount lustre-MDT0000 complete
[10361.825538] LustreError: 216144:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10361.844970] LustreError: 216144:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages
[10362.775118] LustreError: 216125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851835 with bad export cookie 4412907944161803420
[10362.781762] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10362.785039] LustreError: 216125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages
[10363.071680] Lustre: server umount lustre-MDT0001 complete
[10376.304357] Lustre: server umount lustre-OST0000 complete
[10391.994575] Lustre: server umount lustre-OST0001 complete
[10416.171681] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[10435.326289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10436.559552] LustreError: 236083:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10436.830023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10436.874157] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10442.221406] LustreError: 236084:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10444.028720] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10451.430696] LustreError: 236083:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10451.480658] LustreError: 236083:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message
[10455.822526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10456.168730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[10456.202082] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10461.348880] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10464.896916] Lustre: 237231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10472.254062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10472.476484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10472.483856] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10479.290474] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10481.698726] LustreError: 237583:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10481.719274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[10481.727290] LustreError: 237583:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message
[10489.008853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10489.256610] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10489.273488] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[10494.440795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[10495.907729] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10503.671623] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10507.287980] Lustre: 239104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10513.912550] LustreError: 236080:0:(osd_handler.c:3425:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[10519.528592] LustreError: lustre-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -107
[10519.535446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10519.555447] LustreError: Skipped 1 previous similar message
[10519.596266] Lustre: Skipped 1 previous similar message
[10519.609383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10519.615734] Lustre: Skipped 2 previous similar messages
[10524.215645] Lustre: server umount lustre-MDT0000 complete
[10525.153917] LustreError: 236077:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10525.182752] LustreError: 236077:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages
[10528.608179] LustreError: 236063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852001 with bad export cookie 4412907944161852350
[10528.626368] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10528.628305] LustreError: 236063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[10529.080138] Lustre: server umount lustre-MDT0001 complete
[10535.903809] LustreError: 3659:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff9ffb235b8000 x1874687823063936/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[10535.926057] LustreError: 3658:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0000 qtype:usr lqe: ffff9ffa10679200 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 1516 qunit:0 qtune:0 edquot:0 default:no revoke:0
[10535.944230] LustreError: 3658:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 1 previous similar message
[10543.327533] Lustre: server umount lustre-OST0000 complete
[10546.143258] Lustre: 3659:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852003/real 1787852003]  req@ffff9ffa12b48780 x1874687823063296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787852019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[10546.179366] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10546.189227] Lustre: Skipped 3 previous similar messages
[10548.015674] Lustre: server umount lustre-OST0001 complete
[10569.749358] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local
[10580.617751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10581.296550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10586.066854] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10591.712580] LustreError: 241497:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[10591.732890] LustreError: 241497:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages
[10595.167663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[10599.900634] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10603.505357] Lustre: 242645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[10610.846174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10617.719569] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10626.552808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97)
[10630.134408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[10635.769103] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97)
[10638.049177] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10646.036698] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10649.503574] Lustre: 244523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[10679.158281] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 13:35:50 (1787852150)
[10732.651977] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10735.261926] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 13:36:46 (1787852206)
[10762.580370] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10764.251656] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 13:37:16 (1787852236)
[10811.875923] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[10814.455836] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 13:38:05 (1787852285)
[10849.653589] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 13:38:40 (1787852320)
[10868.154567] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[10877.045448] Lustre: 250988:0:(qsd_reint.c:249:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[10882.344793] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[10888.770471] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[10892.557287] Lustre: DEBUG MARKER: Write...
[10914.252113] LustreError: 252335:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[10921.321676] Lustre: DEBUG MARKER: Write...
[10935.187677] Lustre: DEBUG MARKER: Write...
[11011.960913] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 13:41:23 (1787852483)
[11040.032529] LustreError: 256301:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11091.533799] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 13:42:42 (1787852562)
[11121.161757] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11123.396712] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11211.773477] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 13:44:42 (1787852682)
[11264.680205] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 13:45:35 (1787852735)
[11304.404352] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 13:46:15 (1787852775)
[11316.229125] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[11345.969584] Lustre: DEBUG MARKER: Write...
[11348.594469] Lustre: DEBUG MARKER: Write out of block quota ...
[11446.484854] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 13:48:38 (1787852918)
[11459.349673] Lustre: DEBUG MARKER: User quota (block hardlimit:1000 MB)
[11540.633191] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 13:50:11 (1787853011)
[11553.207190] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11575.808026] Lustre: DEBUG MARKER: Write...
[11577.783483] Lustre: DEBUG MARKER: Write out of block quota ...
[11633.396345] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 13:51:44 (1787853104)
[11664.059738] Lustre: DEBUG MARKER: set to use default quota
[11666.050271] Lustre: DEBUG MARKER: set default quota
[11667.695776] Lustre: DEBUG MARKER: get default quota
[11674.538464] Lustre: DEBUG MARKER: Test not out of quota
[11678.628051] Lustre: DEBUG MARKER: Test out of quota
[11690.602274] Lustre: DEBUG MARKER: Increase default quota
[11715.967530] Lustre: DEBUG MARKER: Set quota to override default quota
[11716.024737] LustreError: 244288:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45056 time:1788457989 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11726.737912] Lustre: DEBUG MARKER: Set to use default quota again
[11746.167822] Lustre: DEBUG MARKER: Cleanup
[11805.618389] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 13:54:37 (1787853277)
[11829.133590] Lustre: DEBUG MARKER: set default quota for qpool1
[11830.715687] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11875.150754] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 13:55:46 (1787853346)
[11960.502158] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 13:57:11 (1787853431)
[12038.672349] Lustre: DEBUG MARKER: Write...
[12041.735880] Lustre: DEBUG MARKER: Write out of block quota ...
[12151.979874] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 14:00:23 (1787853623)
[12189.233957] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 14:00:59 (1787853659)
[12197.246481] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 14:01:08 (1787853668)
[12234.863237] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 14:01:46 (1787853706)
[12276.232775] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 14:02:27 (1787853747)
[12299.165908] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 14:02:50 (1787853770)
[12328.673185] Lustre: *** cfs_fail_loc=a06, val=0***
[12329.143310] LustreError: 241510:0:(qmt_lock.c:479:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=2048 : rc = -11
                 qmt:lustre-QMT0000 pool:dt-0x0 id:60000 enforced:1 hard:102400 soft:0 granted:25600 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no
[12331.062917] Lustre: DEBUG MARKER: Files for user (quota_usr), count=5:
[12332.814913] Lustre: DEBUG MARKER: sanity-quota test_80: @@@@@@ FAIL: write failed