[ 5363.180478] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5369.232361] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5372.106793] LustreError: 121878:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5372.966890] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5374.446092] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5376.312376] Lustre: Failing over lustre-MDT0000
[ 5376.659545] Lustre: server umount lustre-MDT0000 complete
[ 5394.920641] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5395.196627] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5395.214368] Lustre: Skipped 2 previous similar messages
[ 5395.353684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5395.408275] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5395.819474] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5395.966380] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5396.016737] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:648 to 0x280000400:673)
[ 5396.018341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:652 to 0x240000400:673)
[ 5396.255078] Lustre: 3320:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779874988/real 1779874988]  req@ffff8c2305c8ad80 x1866328692776960/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1779875004 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5396.294644] Lustre: 3320:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5400.067770] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5400.356695] Lustre: 3322:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779874993/real 1779874993]  req@ffff8c241938f800 x1866328692777728/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779875009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5400.558902] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5400.567973] Lustre: Skipped 1 previous similar message
[ 5404.575344] Lustre: 3321:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779874998/real 1779874998]  req@ffff8c243bd9c380 x1866328692777984/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779875014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5404.607839] Lustre: 3321:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5408.267337] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5409.759130] Lustre: 3320:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779875003/real 1779875003]  req@ffff8c241938c000 x1866328692778496/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779875019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5409.795647] Lustre: 3320:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5410.454185] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5415.328655] Lustre: DEBUG MARKER: (dd_pid=115656, time=0, timeout=600)
[ 5469.684756] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5476.806839] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5480.578891] LustreError: 124712:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5481.908680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5483.847944] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5486.165967] Lustre: Failing over lustre-MDT0000
[ 5486.441867] Lustre: server umount lustre-MDT0000 complete
[ 5502.945324] Lustre: 3322:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779875096/real 1779875096]  req@ffff8c242dfa0380 x1866328692815360/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1779875112 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5502.982704] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5504.033563] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5514.322814] Lustre: 3322:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779875107/real 1779875107]  req@ffff8c2348a4d500 x1866328692816512/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779875123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5514.350573] Lustre: 3322:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 5514.866574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5514.993788] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5516.311515] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5516.570576] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5516.621874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5516.623930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:648 to 0x280000400:705)
[ 5520.085845] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5527.971947] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5527.979645] Lustre: Skipped 1 previous similar message
[ 5528.656424] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5530.754300] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5550.824655] Lustre: DEBUG MARKER: (dd_pid=118142, time=12, timeout=600)
[ 5618.752256] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 05:47:05 (1779875225)
[ 5644.096264] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5646.296356] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5653.577834] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5656.417092] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5703.228757] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 05:48:29 (1779875309)
[ 5729.801510] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5746.868200] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5749.020750] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5751.135946] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5753.730919] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5758.632510] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5762.936725] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5767.615863] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5772.036405] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5775.495680] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5779.524848] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5850.421759] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 05:50:57 (1779875457)
[ 5865.265521] Lustre: server umount lustre-MDT0000 complete
[ 5869.926333] LustreError: 6578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875479 with bad export cookie 145907276420883294
[ 5869.956909] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5880.402520] Lustre: server umount lustre-OST0000 complete
[ 5883.296090] Lustre: 3322:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779875476/real 1779875476]  req@ffff8c242e51b800 x1866328692949632/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779875492 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5883.320712] Lustre: 3322:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5883.334431] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5883.342815] Lustre: Skipped 1 previous similar message
[ 5883.873073] Lustre: server umount lustre-OST0001 complete
[ 5897.972940] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local
[ 5908.569260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5913.215745] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5916.845226] Lustre: 134407:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5923.147434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5930.167319] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5933.347635] LustreError: 134719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5933.376581] LustreError: 134719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 5933.404388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:737)
[ 5938.881247] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 5944.352468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:737)
[ 5946.415848] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5954.881772] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 5958.945269] Lustre: 136197:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 5977.442195] Lustre: server umount lustre-MDT0000 complete
[ 5982.786476] LustreError: 133972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875592 with bad export cookie 145907276420890196
[ 5982.810573] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5993.528195] Lustre: server umount lustre-OST0000 complete
[ 5996.453244] Lustre: 3320:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779875589/real 1779875589]  req@ffff8c2408ef3b80 x1866328692992000/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779875605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5998.316112] Lustre: server umount lustre-OST0001 complete
[ 6015.199434] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local
[ 6024.999303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6030.906820] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6035.228055] Lustre: 138779:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6043.090804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6050.219305] LustreError: 139102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6050.262199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:769)
[ 6050.676324] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6055.402449] LustreError: 139103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6064.640073] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:707 to 0x280000400:769)
[ 6066.407894] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6073.850800] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6077.951680] Lustre: 140574:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6087.337117] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 05:54:54 (1779875694)
[ 6089.207230] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6091.143597] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 05:54:57 (1779875697)
[ 6148.605456] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 05:55:55 (1779875755)
[ 6198.359634] Lustre: DEBUG MARKER: Write...
[ 6201.489230] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6263.206497] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 05:57:49 (1779875869)
[ 6273.373841] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 05:57:59 (1779875879)
[ 6285.505752] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 05:58:11 (1779875891)
[ 6294.661723] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 05:58:21 (1779875901)
[ 6303.083281] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 05:58:29 (1779875909)
[ 6369.411029] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6545.735498] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6756.790362] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 06:06:02 (1779876362)
[ 6823.216220] Lustre: DEBUG MARKER: Restart...
[ 6827.488903] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6827.496580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6827.504916] Lustre: Skipped 2 previous similar messages
[ 6832.609290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6832.612101] Lustre: Skipped 1 previous similar message
[ 6834.043256] Lustre: server umount lustre-MDT0000 complete
[ 6838.946216] LustreError: 142997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876448 with bad export cookie 145907276420891729
[ 6838.965868] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6839.114490] Lustre: server umount lustre-OST0000 complete
[ 6845.558783] Lustre: server umount lustre-OST0001 complete
[ 6867.331331] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local
[ 6878.376996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6878.379838] Lustre: Skipped 1 previous similar message
[ 6884.052145] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6887.757609] Lustre: 158997:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6894.067951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6901.648180] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6903.209270] LustreError: 159321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6903.261517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:779 to 0x240000400:801)
[ 6908.384618] LustreError: 159732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6910.977783] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6916.096908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:777 to 0x280000400:801)
[ 6918.936352] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6927.905949] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6931.985702] Lustre: 160788:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7007.734728] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 06:10:13 (1779876613)
[ 7091.084233] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 06:11:37 (1779876697)
[ 8997.165686] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 06:43:23 (1779878603)
[ 9020.402920] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9020.411398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9020.421802] Lustre: Skipped 1 previous similar message
[ 9020.439661] Lustre: Skipped 1 previous similar message
[ 9024.435972] Lustre: server umount lustre-MDT0000 complete
[ 9029.443115] LustreError: 165601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878638 with bad export cookie 145907276420898225
[ 9029.448119] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9029.451890] LustreError: 165601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 9029.532234] Lustre: server umount lustre-OST0000 complete
[ 9034.577034] Lustre: server umount lustre-OST0001 complete
[ 9049.217941] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local
[ 9059.530094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9064.132412] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9067.400241] Lustre: 168926:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9073.570098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9079.764870] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9083.756497] LustreError: 169246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9083.780553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5803 to 0x240000400:5825)
[ 9087.505852] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9092.619803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5803 to 0x280000400:5825)
[ 9095.081155] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9103.791211] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9107.531568] Lustre: 170713:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9145.590652] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 06:45:52 (1779878752)
[ 9191.593939] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 06:46:38 (1779878798)
[ 9237.826395] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 06:47:24 (1779878844)
[ 9239.863560] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9242.010574] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 06:47:28 (1779878848)
[ 9243.940506] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9246.301735] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 06:47:32 (1779878852)
[ 9335.855882] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 06:49:02 (1779878942)
[ 9387.537031] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 06:49:54 (1779878994)
[ 9509.426781] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted