[ 5698.783100] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5705.499514] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5708.759757] LustreError: 121254:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5709.593320] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5711.512735] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5713.514625] Lustre: Failing over lustre-MDT0000
[ 5713.748749] Lustre: server umount lustre-MDT0000 complete
[ 5730.275406] Lustre: 3309:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385697/real 1787385697]  req@ffff916b0b001880 x1874203856045312/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787385713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5730.275911] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5730.305340] Lustre: 3309:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 5730.345612] Lustre: Skipped 1 previous similar message
[ 5732.282163] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5732.923497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5732.985415] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5734.071693] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5734.218659] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5734.255164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:628 to 0x240000400:673)
[ 5734.265381] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:624 to 0x280000400:641)
[ 5735.523654] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385702/real 1787385702]  req@ffff916bd12f2680 x1874203856045696/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5737.965831] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5740.008351] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5740.176185] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385707/real 1787385707]  req@ffff916bfa1a4700 x1874203856046080/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5740.204632] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5745.824214] Lustre: 3309:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385712/real 1787385712]  req@ffff916bfa1a4e00 x1874203856046336/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5745.857926] Lustre: 3309:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5748.688744] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5750.454455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5761.106892] Lustre: DEBUG MARKER: (dd_pid=114809, time=4, timeout=600)
[ 5809.431992] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5816.452312] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5819.413323] LustreError: 124066:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5820.602820] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5822.778565] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5825.482465] Lustre: Failing over lustre-MDT0000
[ 5825.823357] Lustre: server umount lustre-MDT0000 complete
[ 5841.249435] Lustre: 3308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385808/real 1787385808]  req@ffff916ac475f100 x1874203856085888/t0(0) o601->lustre-MDT0000-lwp-OST0000@0@lo:23/10 lens 336/336 e 0 to 1 dl 1787385824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[ 5841.281061] Lustre: 3308:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5841.289129] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5843.360269] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5854.288829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5854.369675] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5855.222598] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5855.242579] Lustre: Skipped 1 previous similar message
[ 5856.803571] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5857.023894] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5857.065655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:675 to 0x240000400:705)
[ 5857.071249] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:624 to 0x280000400:673)
[ 5859.297495] Lustre: 3308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385826/real 1787385826]  req@ffff916bfa394e00 x1874203856087936/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5859.337047] Lustre: 3308:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 5859.586835] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5869.268988] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5870.991498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5887.825841] Lustre: DEBUG MARKER: (dd_pid=117197, time=8, timeout=600)
[ 5955.510728] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 04:05:37 (1787385937)
[ 5978.206908] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5980.559271] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5987.377425] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5990.204418] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 6023.492590] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 04:06:45 (1787386005)
[ 6052.981567] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 6071.735293] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 6073.528616] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 6075.521397] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 6077.766097] Lustre: DEBUG MARKER: Set quota for 1 times
[ 6081.716946] Lustre: DEBUG MARKER: Set quota for 2 times
[ 6086.457903] Lustre: DEBUG MARKER: Set quota for 3 times
[ 6090.891773] Lustre: DEBUG MARKER: Set quota for 4 times
[ 6096.759810] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6102.267142] Lustre: DEBUG MARKER: Set quota for 6 times
[ 6107.200541] Lustre: DEBUG MARKER: Set quota for 7 times
[ 6148.771501] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 04:08:50 (1787386130)
[ 6163.031550] Lustre: server umount lustre-MDT0000 complete
[ 6168.526676] LustreError: 11221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386151 with bad export cookie 1729969679664306758
[ 6168.540861] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6178.930361] Lustre: server umount lustre-OST0000 complete
[ 6179.819318] Lustre: 3306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787386146/real 1787386146]  req@ffff916b0b1d4380 x1874203856210048/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787386162 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6179.935971] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6179.964970] Lustre: Skipped 1 previous similar message
[ 6185.498930] Lustre: server umount lustre-OST0001 complete
[ 6204.543949] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local
[ 6214.800807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6220.012857] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6223.579404] Lustre: 133522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6232.842557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6239.912277] LustreError: 133841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6239.961933] LustreError: 133841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message
[ 6239.972118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:737)
[ 6240.381570] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6245.345385] LustreError: 133842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6249.110356] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6254.078505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:705)
[ 6257.095302] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6266.722346] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6270.615766] Lustre: 135353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6288.176323] Lustre: server umount lustre-MDT0000 complete
[ 6292.947958] LustreError: 133070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386276 with bad export cookie 1729969679664313002
[ 6292.956094] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6303.297096] Lustre: server umount lustre-OST0000 complete
[ 6306.272938] Lustre: 3308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787386273/real 1787386273]  req@ffff916c00d27800 x1874203856253568/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787386289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6306.305894] Lustre: 3308:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 6306.313715] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6307.727611] Lustre: server umount lustre-OST0001 complete
[ 6326.831753] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local
[ 6337.596958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6342.368640] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6345.621792] Lustre: 137954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6358.729616] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6362.661521] LustreError: 138266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6362.695576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:708 to 0x240000400:769)
[ 6366.830508] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6366.834129] Lustre: Skipped 1 previous similar message
[ 6371.828232] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:675 to 0x280000400:737)
[ 6373.436978] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6381.922228] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6386.086537] Lustre: 139770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6398.218462] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 04:12:59 (1787386379)
[ 6400.171324] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6402.113348] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 04:13:04 (1787386384)
[ 6462.216916] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 04:14:03 (1787386443)
[ 6528.505658] Lustre: DEBUG MARKER: Write...
[ 6532.382187] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6595.391802] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 04:16:17 (1787386577)
[ 6603.567940] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 04:16:24 (1787386584)
[ 6613.764692] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 04:16:35 (1787386595)
[ 6623.257075] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 04:16:45 (1787386605)
[ 6631.808659] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 04:16:53 (1787386613)
[ 6705.708836] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6873.828896] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 7086.860467] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 04:24:28 (1787387068)
[ 7154.584428] Lustre: DEBUG MARKER: Restart...
[ 7160.293085] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7160.295018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7160.313371] Lustre: Skipped 1 previous similar message
[ 7164.933718] Lustre: server umount lustre-MDT0000 complete
[ 7168.955990] LustreError: 137508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387152 with bad export cookie 1729969679664314535
[ 7168.967821] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7168.976424] LustreError: 137508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 7169.100165] Lustre: server umount lustre-OST0000 complete
[ 7173.393744] Lustre: server umount lustre-OST0001 complete
[ 7192.718246] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local
[ 7204.202489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 7207.479626] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7211.146132] Lustre: 158094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 7217.361396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 7223.703870] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7229.604247] LustreError: 158414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7229.652200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:779 to 0x240000400:801)
[ 7232.190307] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7237.633691] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:745 to 0x280000400:769)
[ 7239.842894] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7248.457845] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7252.577949] Lustre: 159921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7326.734867] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 04:28:28 (1787387308)
[ 7408.252859] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 04:29:50 (1787387390)
[ 9331.685389] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 05:01:53 (1787389313)
[ 9350.112849] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9350.114727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 9350.129199] Lustre: Skipped 1 previous similar message
[ 9350.137967] Lustre: Skipped 2 previous similar messages
[ 9352.497753] Lustre: server umount lustre-MDT0000 complete
[ 9356.171538] LustreError: 157647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389339 with bad export cookie 1729969679664321024
[ 9356.177058] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9356.182295] LustreError: 157647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 9356.268151] Lustre: server umount lustre-OST0000 complete
[ 9359.985133] Lustre: server umount lustre-OST0001 complete
[ 9375.671901] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local
[ 9384.497513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9389.292210] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9392.484250] Lustre: 168322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 9397.222914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9403.722917] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9409.442242] LustreError: 168640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 9409.474762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5804 to 0x240000400:5825)
[ 9411.351453] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9416.706462] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5770 to 0x280000400:5793)
[ 9417.339875] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 9425.415963] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 9429.203565] Lustre: 170139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9463.505620] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 05:04:05 (1787389445)
[ 9508.879464] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 05:04:50 (1787389490)
[ 9553.585679] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 05:05:35 (1787389535)
[ 9555.039441] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9556.876563] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 05:05:38 (1787389538)
[ 9558.551300] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9560.567905] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 05:05:42 (1787389542)
[ 9655.782296] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 05:07:18 (1787389638)
[ 9705.919919] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 05:08:07 (1787389687)
[ 9828.137843] Lustre: DEBUG MARKER: sanity-quota test_48: @@@@@@ FAIL: QMT: mds1 qid 60000 is not deleted
[ 9877.309345] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 05:10:59 (1787389859)
[ 9889.143379] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9889.146027] Lustre: Skipped 1 previous similar message
[ 9891.159546] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9891.165472] Lustre: Skipped 95 previous similar messages
[ 9895.193717] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9895.202864] Lustre: Skipped 139 previous similar messages
[ 9903.199656] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9903.204906] Lustre: Skipped 372 previous similar messages
[ 9919.224929] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9919.232878] Lustre: Skipped 620 previous similar messages
[ 9951.227800] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9951.229582] Lustre: Skipped 1497 previous similar messages
[10015.230192] Lustre: *** cfs_fail_loc=a09, val=0***
[10015.231975] Lustre: Skipped 2839 previous similar messages
[10335.347860] Lustre: *** cfs_fail_loc=a08, val=0***
[10335.350522] Lustre: Skipped 2431 previous similar messages
[10335.356704] Lustre: *** cfs_fail_loc=a08, val=0***
[10335.374539] Lustre: Skipped 1 previous similar message
[10687.460886] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10687.468538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10687.497241] Lustre: Skipped 1 previous similar message
[10687.515818] Lustre: Skipped 1 previous similar message
[10691.899727] Lustre: server umount lustre-MDT0000 complete
[10697.034654] LustreError: 167874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390680 with bad export cookie 1729969679666101306
[10697.055965] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10697.783523] Lustre: server umount lustre-OST0000 complete
[10703.540326] Lustre: server umount lustre-OST0001 complete
[10711.076591] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid
[10720.198878] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local
[10732.231581]  vdc: vdc1 vdc9
[10742.873430]  vde: vde1 vde9
[10742.893955]  vde: vde1 vde9
[10753.083897]  vdf: vdf1 vdf9
[10771.297643] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local
[10781.050073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10781.401581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10781.488404] Lustre: lustre-MDT0000: new disk, initializing
[10781.781619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10781.861975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10787.754873] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10793.614830] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10801.074409] Lustre: lustre-OST0000: new disk, initializing
[10801.077361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10801.081102] Lustre: Skipped 1 previous similar message
[10801.190342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10802.886554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10802.912286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10803.225986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10809.784537] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10821.696714] Lustre: lustre-OST0001: new disk, initializing
[10821.702423] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10821.833506] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10824.117231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10824.124686] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10824.295492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10830.706788] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10842.537324] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10846.545462] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10888.599571] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 05:27:50 (1787390870)
[10896.225475] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 05:27:57 (1787390877)
[10948.321537] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 05:28:49 (1787390929)
[11009.584735] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 05:29:51 (1787390991)
[11042.761816] Lustre: DEBUG MARKER: rename directory return 255
[11091.946886] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 05:31:13 (1787391073)
[11128.124875] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 05:31:49 (1787391109)
[11176.019933] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 05:32:37 (1787391157)
[11347.843232] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 05:35:29 (1787391329)
[11391.768576] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 05:36:13 (1787391373)
[11447.747911] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 05:37:08 (1787391428)
[11659.732968] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 05:40:41 (1787391641)
[11661.991351] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[11664.273924] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 05:40:45 (1787391645)
[11733.611350] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[11735.609819] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 05:41:57 (1787391717)
[11775.672061] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[11778.946968] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 05:42:39 (1787391759)
[11841.236081] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11843.260248] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 05:43:45 (1787391825)
[11902.910799] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 05:44:44 (1787391884)
[11904.374574] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11906.261249] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 05:44:48 (1787391888)
[11935.286622] LustreError: 209925:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11986.386288] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 05:46:07 (1787391967)
[12015.744558] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[12018.259661] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[12215.418788] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 05:49:56 (1787392196)
[12271.322700] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 05:50:53 (1787392253)
[12309.603331] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 05:51:31 (1787392291)
[12311.124490] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[12313.013891] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 05:51:34 (1787392294)
[12314.836672] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[12316.847730] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 05:51:38 (1787392298)
[12335.174688] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[12353.702109] Lustre: DEBUG MARKER: Write...
[12356.904276] Lustre: DEBUG MARKER: Write out of block quota ...
[12422.322807] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 05:53:23 (1787392403)
[12452.707870] Lustre: DEBUG MARKER: set to use default quota
[12455.133319] Lustre: DEBUG MARKER: set default quota
[12457.633807] Lustre: DEBUG MARKER: get default quota
[12465.052183] Lustre: DEBUG MARKER: Test not out of quota
[12469.852763] Lustre: DEBUG MARKER: Test out of quota
[12481.705658] Lustre: DEBUG MARKER: Increase default quota
[12513.884795] Lustre: DEBUG MARKER: Set quota to override default quota
[12513.985707] LustreError: 186916:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1787997297 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12527.353608] Lustre: DEBUG MARKER: Set to use default quota again
[12550.407760] Lustre: DEBUG MARKER: Cleanup
[12621.865446] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 05:56:43 (1787392603)
[12645.927257] Lustre: DEBUG MARKER: set default quota for qpool1
[12647.763346] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[12704.416832] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 05:58:06 (1787392686)
[12800.624861] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 05:59:42 (1787392782)
[12888.570529] Lustre: DEBUG MARKER: Write...
[12892.628617] Lustre: DEBUG MARKER: Write out of block quota ...
[13018.531033] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 06:03:20 (1787393000)
[13068.476196] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 06:04:10 (1787393050)
[13076.930939] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 06:04:18 (1787393058)
[13078.562213] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[13080.598866] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 06:04:22 (1787393062)
[13082.257974] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[13083.931783] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 06:04:25 (1787393065)
[13102.555637] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 06:04:44 (1787393084)
[13103.976625] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[13105.483557] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 06:04:47 (1787393087)
[13121.507445] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[13128.032519] LustreError: 234828:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[13128.042644] LustreError: 234828:0:(qmt_pool.c:1413:qmt_pool_recalc()) Skipped 5 previous similar messages
[13130.813090] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.21@tcp (stopping)
[13132.770136] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[13132.779555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[13132.785023] Lustre: Skipped 1 previous similar message
[13135.894599] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.21@tcp (stopping)
[13136.864405] LustreError: 3309:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED  req@ffff916beebc8a80 x1874203865418112/t0(0) o601->lustre-MDT0000-lwp-MDT0000@0@lo:23/10 lens 336/336 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'lquota_wb_lustr.0' uid:0 gid:0 projid:4294967295
[13136.889337] LustreError: 3309:0:(qsd_handler.c:298:qsd_req_completion()) $$$ DQACQ failed with -5, flags:0x8  qsd:lustre-MDT0000 qtype:prj lqe: ffff916ad1d5d080 id:0 enforced:0 granted: 0 pending:0 waiting:0 req:1 usage: 374 qunit:0 qtune:0 edquot:0 default:no revoke:0
[13136.911770] LustreError: 3309:0:(qsd_handler.c:298:qsd_req_completion()) Skipped 3 previous similar messages
[13138.097806] LustreError: 234828:0:(qmt_pool.c:1413:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[13138.405716] Lustre: server umount lustre-MDT0000 complete
[13145.179584] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[13145.847235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[13145.980578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[13145.986526] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[13150.279498] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[13163.115124] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[13163.130267] Lustre: Skipped 1 previous similar message
[13194.709545] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 06:06:16 (1787393176)
[13231.802949] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 06:06:53 (1787393213)
[13266.814632] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 06:07:28 (1787393248)
[13326.965295] Lustre: *** cfs_fail_loc=a08, val=0***
[13326.971967] Lustre: Skipped 8 previous similar messages
[13326.996315] Lustre: *** cfs_fail_loc=a08, val=0***
[13327.000460] Lustre: Skipped 13 previous similar messages
[13447.761136] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 06:10:29 (1787393429)
[13534.256503] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 06:11:56 (1787393516)
[13613.346422] LustreError: 235363:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1787998396 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[13621.487408] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed