[ 5453.629937] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5459.341602] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5462.583207] LustreError: 122103:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5463.437397] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5465.295778] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5467.636546] Lustre: Failing over lustre-MDT0000
[ 5468.050967] Lustre: server umount lustre-MDT0000 complete
[ 5484.515474] Lustre: 3330:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310425/real 1788310425]  req@ffff933590f2ca80 x1875173760432768/t0(0) o400->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788310441 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5484.531599] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5484.545802] Lustre: 3330:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5484.545892] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5485.535121] Lustre: 3331:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310425/real 1788310425]  req@ffff933590f2f480 x1875173760432896/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788310441 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5489.695212] Lustre: 3331:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310430/real 1788310430]  req@ffff933584046680 x1875173760433408/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788310446 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5489.740451] Lustre: 3331:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5494.756789] Lustre: 3330:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310435/real 1788310435]  req@ffff933584044000 x1875173760433920/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788310451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5494.807868] Lustre: 3330:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5495.462768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5495.587270] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5496.421491] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5496.704023] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5496.912498] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5496.950461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:572 to 0x240000400:609)
[ 5496.953505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:569 to 0x280000400:609)
[ 5500.204383] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5510.322398] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5512.150482] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5529.758722] Lustre: DEBUG MARKER: (dd_pid=115008, time=11, timeout=600)
[ 5580.884111] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5588.253554] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5592.161820] LustreError: 125036:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5593.366192] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5595.638510] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5598.154613] Lustre: Failing over lustre-MDT0000
[ 5598.409668] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5598.416920] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5598.443667] Lustre: Skipped 1 previous similar message
[ 5598.452106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5598.654701] Lustre: server umount lustre-MDT0000 complete
[ 5617.440515] Lustre: 3328:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310558/real 1788310558]  req@ffff933590f2c380 x1875173760478208/t0(0) o400->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788310574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5617.464241] Lustre: 3328:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5617.472037] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5627.807717] LustreError: 3327:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff9335ba0d5180 x1875173760481152/t0(0) o250->MGC192.168.203.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5628.533963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5628.619521] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5633.465950] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5634.015319] Lustre: 3328:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788310574/real 1788310574]  req@ffff93348307b100 x1875173760480384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788310590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5634.049762] Lustre: 3328:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 5636.588441] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[ 5636.604738] Lustre: Skipped 1 previous similar message
[ 5640.062596] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5640.185207] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5640.238722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:611 to 0x240000400:641)
[ 5640.238757] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:569 to 0x280000400:641)
[ 5646.181195] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5647.730792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5663.159869] Lustre: DEBUG MARKER: (dd_pid=117464, time=9, timeout=600)
[ 5720.977961] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 20:57:56 (1788310676)
[ 5744.902874] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5746.698805] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5755.909527] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5757.955986] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5791.260287] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 20:59:07 (1788310747)
[ 5815.813461] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5830.602982] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5832.193374] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5833.763615] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5835.688845] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5839.509797] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5843.310361] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5847.610725] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5851.846396] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5855.702139] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5860.244379] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5864.312761] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5923.543760] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 21:01:19 (1788310879)
[ 5937.631866] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5937.632547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5937.646903] Lustre: Skipped 2 previous similar messages
[ 5937.669344] Lustre: Skipped 1 previous similar message
[ 5939.166260] Lustre: server umount lustre-MDT0000 complete
[ 5942.437640] LustreError: 8482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310899 with bad export cookie 7234771260217059283
[ 5942.446502] LustreError: 8482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5942.449830] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5942.514897] Lustre: server umount lustre-OST0000 complete
[ 5945.754843] Lustre: server umount lustre-OST0001 complete
[ 5960.197777] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local
[ 5971.282376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5976.546676] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5980.131890] Lustre: 134783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 5986.910642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 5994.011287] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5996.068436] LustreError: 135152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5996.078169] LustreError: 135152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message
[ 5996.082151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:673)
[ 6001.130352] LustreError: 135152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6001.835419] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6007.284095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:673)
[ 6007.497434] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6014.252420] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6017.330253] Lustre: 136704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6031.428831] Lustre: server umount lustre-MDT0000 complete
[ 6034.930053] LustreError: 137111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310991 with bad export cookie 7234771260217066248
[ 6034.942204] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6045.201779] Lustre: server umount lustre-OST0000 complete
[ 6057.673497] Lustre: server umount lustre-OST0001 complete
[ 6070.806414] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local
[ 6078.416542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6082.891652] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6085.764321] Lustre: 139350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6098.748073] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6103.015782] LustreError: 139723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6103.069254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:643 to 0x240000400:705)
[ 6108.135087] LustreError: 139723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6108.535751] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6108.548458] Lustre: Skipped 1 previous similar message
[ 6113.790652] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:644 to 0x280000400:705)
[ 6115.824904] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6123.311473] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6126.438353] Lustre: 141274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6134.671489] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 21:04:50 (1788311090)
[ 6135.893467] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6137.554360] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 21:04:53 (1788311093)
[ 6191.484937] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 21:05:46 (1788311146)
[ 6238.917278] Lustre: DEBUG MARKER: Write...
[ 6241.470597] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6299.096347] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 21:07:34 (1788311254)
[ 6305.408431] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 21:07:41 (1788311261)
[ 6314.472113] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 21:07:50 (1788311270)
[ 6322.814563] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 21:07:58 (1788311278)
[ 6330.885477] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 21:08:06 (1788311286)
[ 6394.159267] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6548.542614] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6733.143528] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 21:14:48 (1788311688)
[ 6797.337729] Lustre: DEBUG MARKER: Restart...
[ 6802.923452] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6802.956846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6802.960527] Lustre: Skipped 1 previous similar message
[ 6808.033304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6808.036292] Lustre: Skipped 1 previous similar message
[ 6808.466434] Lustre: server umount lustre-MDT0000 complete
[ 6813.157562] LustreError: 144700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311769 with bad export cookie 7234771260217067781
[ 6813.159228] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6813.165190] LustreError: 144700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6813.264381] Lustre: server umount lustre-OST0000 complete
[ 6818.054104] Lustre: server umount lustre-OST0001 complete
[ 6834.258498] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local
[ 6845.626215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6850.008775] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6853.408778] Lustre: 160219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6859.966739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6867.594282] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6871.205173] LustreError: 160582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6871.219757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:715 to 0x240000400:737)
[ 6876.649913] LustreError: 160582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6877.131956] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6882.301316] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:713 to 0x280000400:737)
[ 6884.365411] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6892.422777] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6897.540223] Lustre: 162120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6967.115731] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 21:18:42 (1788311922)
[ 7046.603402] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 21:20:02 (1788312002)
[ 8801.352596] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 21:49:16 (1788313756)
[ 8820.704250] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8820.708322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 8820.714331] Lustre: Skipped 2 previous similar messages
[ 8820.734699] Lustre: Skipped 1 previous similar message
[ 8823.531621] Lustre: server umount lustre-MDT0000 complete
[ 8827.673073] LustreError: 159772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313784 with bad export cookie 7234771260217074277
[ 8827.681219] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8827.688622] LustreError: 159772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 8827.835768] Lustre: server umount lustre-OST0000 complete
[ 8831.837966] Lustre: server umount lustre-OST0001 complete
[ 8847.138557] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local
[ 8857.129603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8861.135719] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8863.952368] Lustre: 170678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8870.635453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8876.632783] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8881.804103] LustreError: 171049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8881.839702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5740 to 0x240000400:5761)
[ 8884.441075] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8889.877932] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5738 to 0x280000400:5761)
[ 8890.598497] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8897.819984] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8901.640327] Lustre: 172589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8937.724603] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 21:51:33 (1788313893)
[ 8982.029606] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 21:52:17 (1788313937)
[ 9026.949405] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 21:53:02 (1788313982)
[ 9028.614267] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9030.772835] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 21:53:06 (1788313986)
[ 9032.268207] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9034.114579] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 21:53:09 (1788313989)
[ 9127.407527] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 21:54:42 (1788314082)
[ 9182.220806] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 21:55:37 (1788314137)
[ 9315.189640] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QSD: ost1 qid 10000 is not deleted