[ 5498.335802] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5504.355710] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5507.979744] LustreError: 108326:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5508.940192] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5510.539412] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5511.867485] Lustre: Failing over lustre-MDT0000
[ 5512.371730] Lustre: server umount lustre-MDT0000 complete
[ 5528.560890] Lustre: 15472:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775768590/real 1775768590]  req@ffff890106b2f800 x1862022668939136/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1775768606 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ll_ost_io00_003.0' uid:0 gid:0 projid:4294967295
[ 5528.589473] Lustre: 15472:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5528.596610] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5530.096868] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5530.984874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5531.076392] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5531.751605] Lustre: 3322:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775768593/real 1775768593]  req@ffff890206b8ea00 x1862022668939520/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775768609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5531.825692] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5532.098788] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5532.159454] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:566 to 0x280000400:609)
[ 5532.162913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:635 to 0x240000400:673)
[ 5533.922956] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5536.703326] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5536.993326] Lustre: 3322:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775768599/real 1775768599]  req@ffff8901474b0700 x1862022668939904/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775768615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5537.013216] Lustre: 3322:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5541.857285] Lustre: 3320:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775768604/real 1775768604]  req@ffff890208acc000 x1862022668940288/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775768620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5541.900642] Lustre: 3320:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5546.876618] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5549.244314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5556.444325] Lustre: DEBUG MARKER: (dd_pid=108730, time=0, timeout=600)
[ 5602.078819] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5609.371806] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5613.468568] LustreError: 110828:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5614.755402] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5616.748745] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5619.278187] Lustre: Failing over lustre-MDT0000
[ 5619.481815] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[ 5619.494778] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5619.519371] Lustre: Skipped 1 previous similar message
[ 5619.528186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5619.712179] Lustre: server umount lustre-MDT0000 complete
[ 5636.066631] Lustre: 111077:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775768698/real 1775768698]  req@ffff890236dd9880 x1862022668979456/t0(0) o101->lustre-MDT0000-lwp-OST0001@0@lo:23/10 lens 456/496 e 0 to 1 dl 1775768714 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'qsd_reint_0.lus.0' uid:0 gid:0 projid:4294967295
[ 5636.101024] Lustre: 111077:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5637.217163] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5647.464590] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4b56f76c5c6b57bf
[ 5647.486809] Lustre: MGC192.168.206.151@tcp: Connection restored to 0@lo (at 0@lo)
[ 5647.489909] Lustre: Skipped 1 previous similar message
[ 5648.343385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5648.573319] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5650.490591] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5650.693623] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5650.741323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:566 to 0x280000400:641)
[ 5650.743774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5653.915375] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5661.482805] LustreError: 3319:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-lwp-OST0001: namespace resource [0x200000006:0x20000:0x0].0x0 (ffff89020d1a8b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 5661.496662] LustreError: 3319:0:(ldlm_resource.c:1170:ldlm_resource_complain()) Skipped 5 previous similar messages
[ 5662.068663] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5664.439590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5678.826397] Lustre: DEBUG MARKER: (dd_pid=111012, time=7, timeout=600)
[ 5743.470305] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 17:06:59 (1775768819)
[ 5767.504292] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5769.490166] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5776.926848] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5779.185705] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5812.539939] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 17:08:08 (1775768888)
[ 5841.180711] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5859.539670] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5861.114215] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5862.909472] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5865.368447] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5869.443575] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5874.227333] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5878.085496] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5881.481615] Lustre: DEBUG MARKER: Set quota for 5 times
[ 5885.548080] Lustre: DEBUG MARKER: Set quota for 6 times
[ 5888.818112] Lustre: DEBUG MARKER: Set quota for 7 times
[ 5892.330985] Lustre: DEBUG MARKER: Set quota for 8 times
[ 5945.699929] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 17:10:21 (1775769021)
[ 5958.624791] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5958.628506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 5958.657918] Lustre: Skipped 2 previous similar messages
[ 5958.673564] Lustre: Skipped 1 previous similar message
[ 5959.522678] Lustre: server umount lustre-MDT0000 complete
[ 5964.786533] LustreError: 87597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775769043 with bad export cookie 5428798445620778943
[ 5964.787251] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5964.810338] LustreError: 87597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 5964.946495] Lustre: server umount lustre-OST0000 complete
[ 5969.117058] Lustre: server umount lustre-OST0001 complete
[ 5983.664775] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local
[ 5992.182972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5996.251768] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5999.534165] Lustre: 119547:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6006.005618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6013.011183] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6017.252595] LustreError: 119859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6017.282813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:737)
[ 6017.284447] LustreError: 119859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 6021.696160] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6023.199448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:643 to 0x280000400:673)
[ 6029.189888] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6036.119970] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6039.426209] Lustre: 121331:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6062.561354] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6062.564380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6062.579714] Lustre: Skipped 1 previous similar message
[ 6062.594391] Lustre: Skipped 1 previous similar message
[ 6066.226441] Lustre: server umount lustre-MDT0000 complete
[ 6070.469888] LustreError: 119108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775769148 with bad export cookie 5428798445620786097
[ 6070.475652] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6070.490850] LustreError: 119108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 6070.596449] Lustre: server umount lustre-OST0000 complete
[ 6075.057859] Lustre: server umount lustre-OST0001 complete
[ 6088.996196] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local
[ 6098.108291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6102.168505] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6104.823916] Lustre: 123897:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6116.960506] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6122.920412] LustreError: 124217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6122.984161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:769)
[ 6124.230935] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6124.234268] Lustre: Skipped 1 previous similar message
[ 6125.659774] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:643 to 0x280000400:705)
[ 6130.233350] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6139.411475] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6143.633355] Lustre: 125690:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6160.511650] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 17:13:56 (1775769236)
[ 6162.597865] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6164.560170] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 17:14:00 (1775769240)
[ 6214.917541] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 17:14:50 (1775769290)
[ 6259.824517] Lustre: DEBUG MARKER: Write...
[ 6262.506690] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6313.394824] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 17:16:29 (1775769389)
[ 6319.145817] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 17:16:35 (1775769395)
[ 6327.725869] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 17:16:44 (1775769404)
[ 6334.848670] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 17:16:51 (1775769411)
[ 6341.781737] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 17:16:58 (1775769418)
[ 6398.902023] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6537.439932] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6712.559588] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 17:23:09 (1775769789)
[ 6780.700345] Lustre: DEBUG MARKER: Restart...
[ 6784.673562] Lustre: server umount lustre-MDT0000 complete
[ 6788.801878] LustreError: 125692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775769867 with bad export cookie 5428798445620787630
[ 6788.810660] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6799.470046] Lustre: server umount lustre-OST0000 complete
[ 6801.386491] Lustre: 3320:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775769863/real 1775769863]  req@ffff89020995ed80 x1862022669369984/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775769879 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6801.424541] Lustre: 3320:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 6801.431921] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6803.610891] Lustre: server umount lustre-OST0001 complete
[ 6818.924362] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local
[ 6828.301318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6832.541283] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6835.494741] Lustre: 142305:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6840.614077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6846.703596] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6853.802875] LustreError: 142625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6853.848244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:778 to 0x240000400:801)
[ 6855.353584] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6857.010144] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:714 to 0x280000400:737)
[ 6861.917536] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6871.511679] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6875.769559] Lustre: 144098:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6949.290539] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 17:27:04 (1775770024)
[ 7019.458908] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 17:28:15 (1775770095)
[ 8784.830924] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 17:57:40 (1775771860)
[ 8800.124817] Lustre: server umount lustre-MDT0000 complete
[ 8804.460996] LustreError: 141875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775771882 with bad export cookie 5428798445620794119
[ 8804.469951] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8814.184861] Lustre: server umount lustre-OST0000 complete
[ 8817.127849] Lustre: 3321:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775771879/real 1775771879]  req@ffff89014c0c6d80 x1862022672390528/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775771895 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 8817.169971] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8819.674974] Lustre: server umount lustre-OST0001 complete
[ 8837.430691] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local
[ 8848.528439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8853.544767] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8857.333415] Lustre: 151776:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8865.374805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8873.655723] LustreError: 152098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8873.783391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5804 to 0x240000400:5825)
[ 8875.028415] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8879.083270] LustreError: 152098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8884.203768] LustreError: 152098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8886.027537] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8887.675252] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5738 to 0x280000400:5761)
[ 8894.954458] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8904.539779] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8908.423558] Lustre: 153578:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 8941.346552] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 18:00:17 (1775772017)
[ 8986.366973] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 18:01:01 (1775772061)
[ 9032.359377] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 18:01:48 (1775772108)
[ 9033.989533] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9037.205658] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 18:01:52 (1775772112)
[ 9039.443972] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9041.541536] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 18:01:57 (1775772117)
[ 9133.129978] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 18:03:28 (1775772208)
[ 9180.190983] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 18:04:15 (1775772255)
[ 9271.072746] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 18:05:46 (1775772346)
[ 9281.832767] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9281.834089] Lustre: Skipped 2 previous similar messages
[ 9283.854297] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9283.856322] Lustre: Skipped 79 previous similar messages
[ 9287.879374] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9287.882015] Lustre: Skipped 179 previous similar messages
[ 9295.889044] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9295.895045] Lustre: Skipped 323 previous similar messages
[ 9311.904873] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9311.908884] Lustre: Skipped 673 previous similar messages
[ 9343.919532] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9343.931245] Lustre: Skipped 1349 previous similar messages
[ 9407.954176] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9407.957574] Lustre: Skipped 2693 previous similar messages
[ 9823.911950] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9823.921802] Lustre: Skipped 2700 previous similar messages
[ 9823.930341] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9823.940279] Lustre: Skipped 2 previous similar messages
[10114.727823] LustreError: lustre-MDT0000-lwp-OST0000: operation quota_acquire to node 0@lo failed: rc = -107
[10114.729350] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10114.737490] LustreError: Skipped 1 previous similar message
[10114.764273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10116.065120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10116.073703] Lustre: Skipped 2 previous similar messages
[10117.985541] Lustre: server umount lustre-MDT0000 complete
[10121.921894] LustreError: 153581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775773200 with bad export cookie 5428798445622573512
[10121.947548] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10122.308453] Lustre: server umount lustre-OST0000 complete
[10126.608782] Lustre: server umount lustre-OST0001 complete
[10132.611879] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid
[10141.226970] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local
[10154.758149]  vdc: vdc1 vdc9
[10164.141971]  vde: vde1 vde9
[10174.346915]  vdf: vdf1 vdf9
[10191.073194] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local
[10203.982888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10204.363476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10204.424743] Lustre: lustre-MDT0000: new disk, initializing
[10205.026911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10205.132035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10210.852208] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10216.170838] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10222.307216] Lustre: lustre-OST0000: new disk, initializing
[10222.315261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10222.321285] Lustre: Skipped 1 previous similar message
[10222.393640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10223.783300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10223.787872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10223.980511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10228.533295] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10238.213909] Lustre: lustre-OST0001: new disk, initializing
[10238.220651] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10238.313250] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10239.814474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10239.820744] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10239.890504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10245.176649] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10256.208505] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10261.572138] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10307.844159] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 18:23:03 (1775773383)
[10314.708750] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 18:23:10 (1775773390)
[10357.720873] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 18:23:53 (1775773433)
[10419.977326] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 18:24:55 (1775773495)
[10450.197568] Lustre: DEBUG MARKER: rename directory return 255
[10495.702818] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 18:26:11 (1775773571)
[10525.746585] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 18:26:41 (1775773601)
[10570.486978] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 18:27:26 (1775773646)
[10671.920824] LustreError: 170201:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10715.706643] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 18:29:50 (1775773790)
[10750.007667] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 18:30:26 (1775773826)
[10810.447041] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 18:31:25 (1775773885)
[11005.964939] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 18:34:41 (1775774081)
[11008.155915] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11010.760907] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 18:34:46 (1775774086)
[11075.581815] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11078.429226] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 18:35:53 (1775774153)
[11108.431391] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11110.024202] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 18:36:26 (1775774186)
[11162.442715] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11164.536102] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 18:37:20 (1775774240)
[11213.882365] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 18:38:09 (1775774289)
[11215.485793] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11217.407605] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 18:38:13 (1775774293)
[11244.904562] LustreError: 186883:0:(mgs_handler.c:1102:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11289.352692] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 18:39:25 (1775774365)
[11314.646225] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11316.328770] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11553.096409] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 18:43:48 (1775774628)
[11602.349356] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 18:44:38 (1775774678)
[11644.114973] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 18:45:19 (1775774719)
[11646.739301] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11650.240932] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 18:45:24 (1775774724)
[11652.437649] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11655.123781] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 18:45:30 (1775774730)
[11671.749715] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11689.182488] Lustre: DEBUG MARKER: Write...
[11691.744300] Lustre: DEBUG MARKER: Write out of block quota ...
[11751.725660] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 18:47:07 (1775774827)
[11780.206979] Lustre: DEBUG MARKER: set to use default quota
[11782.073453] Lustre: DEBUG MARKER: set default quota
[11783.682528] Lustre: DEBUG MARKER: get default quota
[11790.253605] Lustre: DEBUG MARKER: Test not out of quota
[11794.870598] Lustre: DEBUG MARKER: Test out of quota
[11807.622471] Lustre: DEBUG MARKER: Increase default quota
[11842.378493] Lustre: DEBUG MARKER: Set quota to override default quota
[11842.445580] LustreError: 166795:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1776379720 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11854.683438] Lustre: DEBUG MARKER: Set to use default quota again
[11878.360685] Lustre: DEBUG MARKER: Cleanup
[11948.616807] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 18:50:24 (1775775024)
[11973.724464] Lustre: DEBUG MARKER: set default quota for qpool1
[11975.308447] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12027.273601] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 18:51:43 (1775775103)
[12116.790155] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 18:53:12 (1775775192)
[12199.536426] Lustre: DEBUG MARKER: Write...
[12203.218876] Lustre: DEBUG MARKER: Write out of block quota ...
[12330.766919] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 18:56:46 (1775775406)
[12374.778633] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 18:57:30 (1775775450)
[12381.847974] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 18:57:37 (1775775457)
[12383.128471] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12384.795736] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 18:57:40 (1775775460)
[12386.454787] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12387.950437] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 18:57:44 (1775775464)
[12406.857714] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 18:58:03 (1775775483)
[12408.495238] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12410.605488] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 18:58:06 (1775775486)
[12426.541658] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12433.679338] LustreError: 209600:0:(qmt_pool.c:1399:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12433.691261] LustreError: 209600:0:(qmt_pool.c:1399:qmt_pool_recalc()) Skipped 5 previous similar messages
[12436.449144] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12436.465935] Lustre: Skipped 1 previous similar message
[12436.473080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12436.481310] Lustre: Skipped 1 previous similar message
[12441.574262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12441.584595] Lustre: Skipped 2 previous similar messages
[12443.769090] LustreError: 209600:0:(qmt_pool.c:1399:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12444.159898] Lustre: server umount lustre-MDT0000 complete
[12451.408344] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12452.002608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12452.162465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12452.164402] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12456.495140] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12466.691698] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12466.704948] Lustre: Skipped 2 previous similar messages
[12499.853313] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 18:59:35 (1775775575)
[12531.321740] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 19:00:07 (1775775607)
[12559.989299] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 19:00:36 (1775775636)
[12617.421145] Lustre: *** cfs_fail_loc=a08, val=0***
[12617.432213] Lustre: Skipped 6 previous similar messages
[12617.454588] Lustre: *** cfs_fail_loc=a08, val=0***
[12617.459220] Lustre: Skipped 13 previous similar messages
[12737.771074] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 19:03:34 (1775775814)
[12819.626218] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 19:04:55 (1775775895)
[12897.774402] LustreError: 217679:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1776380775 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13058.558611] LustreError: 210134:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1776380936 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13214.045689] LustreError: 217679:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:1000 enforced:1 hard:2048 soft:2048 granted:65536 time:1776381092 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13214.129917] LustreError: 210175:0:(qsd_reint.c:627:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:1 until pending updates are flushed.
[13222.438601] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed