[ 5534.069494] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5538.454952] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5540.813268] LustreError: 120411:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5541.497174] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5542.650780] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5544.693280] Lustre: Failing over lustre-MDT0000
[ 5545.084146] Lustre: server umount lustre-MDT0000 complete
[ 5560.866750] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5561.080227] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5561.091164] Lustre: Skipped 1 previous similar message
[ 5561.225377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5561.300283] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5561.888200] Lustre: 3296:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243534/real 1781243534]  req@ffff93df4a7043c0 x1867763461477120/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5561.915454] Lustre: 3296:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5562.719150] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5562.825229] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5562.846556] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:664 to 0x280000400:705)
[ 5562.846844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:669 to 0x240000400:705)
[ 5564.023371] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5566.444545] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5567.458185] Lustre: 3295:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243539/real 1781243539]  req@ffff93df4d550f00 x1867763461477760/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5567.481809] Lustre: 3295:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5569.784072] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5571.010418] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5571.552484] Lustre: 3298:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243544/real 1781243544]  req@ffff93df4d550000 x1867763461478016/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5571.586513] Lustre: 3298:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5576.216623] Lustre: DEBUG MARKER: (dd_pid=114497, time=1, timeout=600)
[ 5617.210634] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5621.880903] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5624.564643] LustreError: 123201:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5625.471150] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5626.915316] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5628.685073] Lustre: Failing over lustre-MDT0000
[ 5628.985766] Lustre: server umount lustre-MDT0000 complete
[ 5644.642975] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5644.939318] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5645.127953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5645.202686] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5646.048397] Lustre: 3298:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243618/real 1781243618]  req@ffff93e0379012c0 x1867763461510400/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1781243634 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5646.064226] Lustre: 3298:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5648.259626] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5650.406357] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5650.416028] Lustre: Skipped 1 previous similar message
[ 5651.304697] LustreError: 124186:0:(tgt_handler.c:533:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffff93df02c18b40 x1867763461523200/t0(0) o601->lustre-MDT0000-lwp-OST0000_UUID@0@lo:85/0 lens 336/0 e 0 to 0 dl 1781243650 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295
[ 5651.318788] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -11
[ 5654.112172] Lustre: 3296:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243626/real 1781243626]  req@ffff93df0459a580 x1867763461511168/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5654.138031] Lustre: 3296:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5654.320174] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5654.412586] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5654.464060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:707 to 0x240000400:737)
[ 5654.467070] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:664 to 0x280000400:737)
[ 5657.597024] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5659.197278] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5663.479667] Lustre: DEBUG MARKER: (dd_pid=116947, time=0, timeout=600)
[ 5712.177509] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 01:54:58 (1781243698)
[ 5732.314543] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5737.810566] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5748.082461] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5751.455859] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5805.232290] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 01:56:31 (1781243791)
[ 5834.951416] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5851.129792] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5852.837464] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5854.422448] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5856.428757] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5860.186706] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5864.085561] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5868.072880] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5871.825822] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5875.624109] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5880.079515] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5883.973796] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5947.398506] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 01:58:53 (1781243933)
[ 5961.211360] Lustre: server umount lustre-MDT0000 complete
[ 5965.759492] LustreError: 8237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243954 with bad export cookie 2947257502305830006
[ 5965.766885] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5976.184560] Lustre: server umount lustre-OST0000 complete
[ 5991.149313] Lustre: server umount lustre-OST0001 complete
[ 6005.375873] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local
[ 6015.075672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6020.363796] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6023.594670] Lustre: 132732:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6029.683179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6036.480337] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6039.911916] LustreError: 133048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6039.941624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:769)
[ 6045.104475] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6050.347970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:740 to 0x280000400:769)
[ 6055.512500] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6065.958503] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6070.789401] Lustre: 134520:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6090.111503] Lustre: server umount lustre-MDT0000 complete
[ 6095.178503] LustreError: 132296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244083 with bad export cookie 2947257502305837055
[ 6095.202920] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6104.725177] Lustre: server umount lustre-OST0000 complete
[ 6106.528899] Lustre: 3297:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244079/real 1781244079]  req@ffff93e03c69a940 x1867763461678848/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781244095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6106.562514] Lustre: 3297:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 6106.576273] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6106.603857] Lustre: Skipped 1 previous similar message
[ 6109.829722] Lustre: server umount lustre-OST0001 complete
[ 6130.069809] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local
[ 6142.729699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6148.670687] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6152.326967] Lustre: 137157:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6158.873402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6164.871733] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6167.022180] LustreError: 137460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6167.051279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:739 to 0x240000400:801)
[ 6172.130498] LustreError: 137925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6178.834139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:740 to 0x280000400:801)
[ 6180.496932] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6189.682565] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6194.487188] Lustre: 138928:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6207.320268] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 02:03:13 (1781244193)
[ 6208.948309] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6211.185620] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 02:03:17 (1781244197)
[ 6269.433242] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 02:04:14 (1781244254)
[ 6319.712706] Lustre: DEBUG MARKER: Write...
[ 6323.199235] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6382.025316] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 02:06:07 (1781244367)
[ 6393.081282] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 02:06:18 (1781244378)
[ 6404.354751] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 02:06:30 (1781244390)
[ 6414.569464] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 02:06:40 (1781244400)
[ 6424.068702] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 02:06:49 (1781244409)
[ 6492.972079] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6660.822356] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6857.968878] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 02:14:04 (1781244844)
[ 6923.758158] Lustre: DEBUG MARKER: Restart...
[ 6931.219313] Lustre: server umount lustre-MDT0000 complete
[ 6936.543227] LustreError: 136730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244925 with bad export cookie 2947257502305838588
[ 6936.552767] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6946.862028] Lustre: server umount lustre-OST0000 complete
[ 6947.816829] Lustre: 3295:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244920/real 1781244920]  req@ffff93e02efd83c0 x1867763461911296/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781244936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6947.840897] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6951.279646] Lustre: server umount lustre-OST0001 complete
[ 6968.583241] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local
[ 6981.959668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6981.966542] Lustre: Skipped 1 previous similar message
[ 6987.165639] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6990.334444] Lustre: 157457:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6995.922855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7004.351761] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7007.152426] LustreError: 157775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7007.191261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:810 to 0x240000400:833)
[ 7012.324096] LustreError: 157775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7014.489037] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7019.511311] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:810 to 0x280000400:833)
[ 7022.676914] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7032.040293] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7036.016496] Lustre: 159249:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7109.125709] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 02:18:14 (1781245094)
[ 7188.508754] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 02:19:33 (1781245173)
[ 9031.591729] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 02:50:16 (1781247016)
[ 9050.596377] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9050.601704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9050.617954] Lustre: Skipped 1 previous similar message
[ 9050.634960] Lustre: Skipped 1 previous similar message
[ 9050.834523] Lustre: server umount lustre-MDT0000 complete
[ 9057.409698] LustreError: 157026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247045 with bad export cookie 2947257502305845084
[ 9057.415936] LustreError: MGC192.168.202.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9057.439232] LustreError: 157026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 9057.659027] Lustre: server umount lustre-OST0000 complete
[ 9064.056933] Lustre: server umount lustre-OST0001 complete
[ 9082.636529] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing load_modules_local
[ 9096.021281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9102.314130] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9106.086069] Lustre: 167536:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9113.840099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9120.651195] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9121.965318] LustreError: 167851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9121.997139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5835 to 0x240000400:5857)
[ 9127.393579] LustreError: 167851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9131.496056] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9136.647191] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5835 to 0x280000400:5857)
[ 9140.448833] Lustre: DEBUG MARKER: oleg203-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9149.130522] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9153.873385] Lustre: 169309:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9195.064142] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 02:53:00 (1781247180)
[ 9249.712680] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 02:53:55 (1781247235)
[ 9297.859176] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 02:54:43 (1781247283)
[ 9299.890799] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9301.757558] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 02:54:47 (1781247287)
[ 9304.080794] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9306.478346] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 02:54:52 (1781247292)
[ 9405.213933] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 02:56:31 (1781247391)
[ 9459.527972] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 02:57:25 (1781247445)
[ 9581.066694] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted