[ 5841.123199] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5848.916498] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5853.475604] LustreError: 120288:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5855.052469] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5857.483651] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5859.902337] Lustre: Failing over lustre-MDT0000
[ 5860.476350] Lustre: server umount lustre-MDT0000 complete
[ 5876.192302] Lustre: 3341:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698778/real 1779698778]  req@ffff8993b51cf100 x1866143435539200/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1779698794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5876.225942] Lustre: 3341:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5876.231385] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5878.177388] Lustre: 3338:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698780/real 1779698780]  req@ffff89928da72a00 x1866143435539328/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779698796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5878.227615] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5884.383901] Lustre: 3338:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698786/real 1779698786]  req@ffff8993b51cca80 x1866143435539968/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779698802 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5884.405187] Lustre: 3338:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5889.305267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5889.418585] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5890.048212] Lustre: 3341:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698791/real 1779698791]  req@ffff89928da71500 x1866143435540224/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779698807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5890.079496] Lustre: 3341:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5890.239222] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5890.426318] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5890.712816] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5890.774360] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:507 to 0x280000400:545)
[ 5890.775399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:511 to 0x240000400:545)
[ 5894.749646] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5903.382041] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5905.263920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5919.145848] Lustre: DEBUG MARKER: (dd_pid=114511, time=7, timeout=600)
[ 5975.686989] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5985.057669] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5988.946957] LustreError: 123005:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5989.923642] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5991.738666] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5994.169379] Lustre: Failing over lustre-MDT0000
[ 5994.319583] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5994.329622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5994.340306] Lustre: Skipped 1 previous similar message
[ 5994.348528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5994.694113] Lustre: server umount lustre-MDT0000 complete
[ 6014.117321] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6014.432118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up)
[ 6014.435795] Lustre: Skipped 1 previous similar message
[ 6014.705594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6014.794705] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6015.526192] Lustre: 3338:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779698917/real 1779698917]  req@ffff899287516300 x1866143435585792/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779698933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6015.577414] Lustre: 3338:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 6015.591455] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 6015.603720] Lustre: Skipped 1 previous similar message
[ 6015.643429] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 6015.877609] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 6015.936747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:547 to 0x240000400:577)
[ 6015.936964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:507 to 0x280000400:577)
[ 6020.148930] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6029.296842] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6031.681278] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6041.374905] Lustre: DEBUG MARKER: (dd_pid=116897, time=3, timeout=600)
[ 6109.758740] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:50:26 (1779699026)
[ 6137.175704] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6140.077562] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6148.681971] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 6151.426201] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6192.444108] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:51:49 (1779699109)
[ 6221.352121] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6237.690169] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6239.592596] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6241.587389] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6244.246986] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6249.027541] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6253.207717] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6257.131367] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6261.587021] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6266.322791] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6270.400717] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6331.934822] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:54:08 (1779699248)
[ 6345.919482] Lustre: server umount lustre-MDT0000 complete
[ 6351.030802] LustreError: 7465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699269 with bad export cookie 6890085486686715466
[ 6351.048761] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6361.654683] Lustre: server umount lustre-OST0000 complete
[ 6364.961156] Lustre: 3338:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779699266/real 1779699266]  req@ffff899390d11880 x1866143435713024/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779699282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6365.004989] Lustre: 3338:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6365.024936] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6365.055178] Lustre: Skipped 1 previous similar message
[ 6366.397222] Lustre: server umount lustre-OST0001 complete
[ 6382.367420] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local
[ 6392.979978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6398.241190] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6401.739226] Lustre: 132324:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6408.021241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6415.475898] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6417.186468] LustreError: 132640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6417.198655] LustreError: 132640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 6417.230763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:580 to 0x240000400:609)
[ 6422.499395] LustreError: 132640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6423.434518] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6428.657047] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:579 to 0x280000400:609)
[ 6429.515761] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6437.987490] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6442.444598] Lustre: 134108:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6463.490995] Lustre: server umount lustre-MDT0000 complete
[ 6468.965825] LustreError: 131889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699386 with bad export cookie 6890085486686722347
[ 6468.984357] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6478.326972] Lustre: server umount lustre-OST0000 complete
[ 6480.226090] Lustre: 3338:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779699382/real 1779699382]  req@ffff8992ceaad500 x1866143435755776/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779699398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6480.260134] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6483.057199] Lustre: server umount lustre-OST0001 complete
[ 6499.219064] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local
[ 6508.875612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6514.142446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6517.440281] Lustre: 136675:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6533.051168] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6533.665457] LustreError: 136999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6533.686779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:580 to 0x240000400:641)
[ 6538.723765] LustreError: 136999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6541.731546] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6541.736971] Lustre: Skipped 1 previous similar message
[ 6546.942174] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:579 to 0x280000400:641)
[ 6548.881277] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6557.912596] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6562.350331] Lustre: 138476:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6574.462145] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:58:11 (1779699491)
[ 6576.436233] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6578.813778] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:58:15 (1779699495)
[ 6635.892790] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:59:12 (1779699552)
[ 6689.297649] Lustre: DEBUG MARKER: Write...
[ 6691.798400] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6749.083248] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 05:01:05 (1779699665)
[ 6756.343976] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 05:01:13 (1779699673)
[ 6767.787927] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 05:01:24 (1779699684)
[ 6777.593742] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 05:01:34 (1779699694)
[ 6788.491900] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 05:01:45 (1779699705)
[ 6854.333483] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 7016.882034] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7205.111629] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 05:08:41 (1779700121)
[ 7271.146683] Lustre: DEBUG MARKER: Restart...
[ 7275.818305] Lustre: server umount lustre-MDT0000 complete
[ 7279.490831] LustreError: 136246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700197 with bad export cookie 6890085486686723880
[ 7279.497273] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7289.916608] Lustre: server umount lustre-OST0000 complete
[ 7293.972239] LustreError: 3340:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-OST0001 qtype:usr lqe: ffff8992857e6900 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 2377 qunit:0 qtune:0 edquot:0 default:no revoke:0
[ 7303.341411] Lustre: server umount lustre-OST0001 complete
[ 7319.563662] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local
[ 7328.250687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7331.946543] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7334.166482] Lustre: 156897:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7338.464684] LustreError: 157218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7338.662132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7343.574087] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7344.105295] LustreError: 157220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7349.219289] LustreError: 157218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7351.037989] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7352.049842] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:650 to 0x280000400:673)
[ 7352.055578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:650 to 0x240000400:673)
[ 7357.268237] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7364.642900] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7368.170736] Lustre: 158684:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7443.661992] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 05:12:40 (1779700360)
[ 7517.174133] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 05:13:54 (1779700434)
[ 9371.365350] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 05:44:48 (1779702288)
[ 9385.541326] Lustre: server umount lustre-MDT0000 complete
[ 9389.181137] LustreError: 156470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702307 with bad export cookie 6890085486686730369
[ 9389.196613] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9399.890509] Lustre: server umount lustre-OST0000 complete
[ 9414.011118] Lustre: server umount lustre-OST0001 complete
[ 9429.262907] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local
[ 9438.186495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9441.939475] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9444.833848] Lustre: 167231:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9450.301656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9458.336185] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9463.546057] LustreError: 167548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9463.607481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5675 to 0x240000400:5697)
[ 9469.030195] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9474.066485] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5675 to 0x280000400:5697)
[ 9476.636433] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9485.045624] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9489.894855] Lustre: 169009:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9528.798191] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 05:47:25 (1779702445)
[ 9574.267124] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 05:48:11 (1779702491)
[ 9620.332784] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 05:48:57 (1779702537)
[ 9622.233174] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9624.605990] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 05:49:01 (1779702541)
[ 9626.824993] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9628.504019] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 05:49:05 (1779702545)
[ 9725.470456] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 05:50:42 (1779702642)
[ 9738.737250] LustreError: 3338:0:(qsd_reint.c:627:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 9738.759903] LustreError: 3338:0:(qsd_reint.c:627:qqi_reint_delayed()) Skipped 2 previous similar messages
[ 9781.612545] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 05:51:38 (1779702698)
[ 9901.616718] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted