[ 5583.978982] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5589.508233] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 5592.709178] LustreError: 121839:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5593.813073] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5596.305176] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5598.784479] Lustre: Failing over lustre-MDT0000
[ 5599.064527] Lustre: server umount lustre-MDT0000 complete
[ 5616.479133] Lustre: 3314:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786170401/real 1786170401]  req@ffff98a288e6c000 x1872929669303424/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786170417 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5616.517394] Lustre: 3314:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5616.528600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5616.543176] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5616.550401] Lustre: Skipped 2 previous similar messages
[ 5621.919135] Lustre: 3315:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786170406/real 1786170406]  req@ffff98a28355b100 x1872929669303808/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786170422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5621.935237] Lustre: 3315:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 5626.871729] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a37b1796ef3624
[ 5626.885208] Lustre: MGC192.168.203.123@tcp: Connection restored to 0@lo (at 0@lo)
[ 5626.897037] Lustre: Skipped 1 previous similar message
[ 5627.282414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5627.349846] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5627.935135] Lustre: 3313:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786170412/real 1786170412]  req@ffff98a2a98c1f80 x1872929669304064/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786170428 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5627.954188] Lustre: 3313:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[ 5632.091629] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5639.973954] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5640.055849] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5640.079864] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:486 to 0x280000400:513)
[ 5640.081793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:490 to 0x240000400:513)
[ 5646.972392] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5648.552379] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5660.351709] Lustre: DEBUG MARKER: (dd_pid=114708, time=6, timeout=600)
[ 5707.313711] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 5713.177566] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 5717.297656] LustreError: 124772:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5718.464541] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5721.394391] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 5723.800834] Lustre: Failing over lustre-MDT0000
[ 5724.198200] Lustre: server umount lustre-MDT0000 complete
[ 5739.487136] Lustre: 3313:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786170524/real 1786170524]  req@ffff98a2bfcdbb80 x1872929669344384/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786170540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5739.488026] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5739.513593] Lustre: 3313:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5739.513718] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5739.535504] Lustre: Skipped 1 previous similar message
[ 5749.791433] Lustre: 3313:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786170534/real 1786170534]  req@ffff98a2871b9880 x1872929669345280/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786170550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 5749.843805] Lustre: 3313:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 5750.236940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5750.289841] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5752.678566] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5752.946167] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5752.990384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:515 to 0x240000400:545)
[ 5752.992305] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:486 to 0x280000400:545)
[ 5754.283623] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 5763.498417] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
[ 5764.449776] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo)
[ 5764.454849] Lustre: Skipped 2 previous similar messages
[ 5765.196627] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5786.104679] Lustre: DEBUG MARKER: (dd_pid=117170, time=14, timeout=600)
[ 5849.976552] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 02:30:50 (1786170650)
[ 5871.643469] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5873.388992] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5879.762971] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 5881.495901] Lustre: DEBUG MARKER: Block quota isn't 0 (u:quota_usr:2).
[ 5916.438066] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 02:31:56 (1786170716)
[ 5942.024391] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 5959.359798] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 5961.249276] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 5963.224185] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 5966.455586] Lustre: DEBUG MARKER: Set quota for 1 times
[ 5972.741276] Lustre: DEBUG MARKER: Set quota for 2 times
[ 5978.664506] Lustre: DEBUG MARKER: Set quota for 3 times
[ 5983.813729] Lustre: DEBUG MARKER: Set quota for 4 times
[ 5989.823737] Lustre: DEBUG MARKER: Set quota for 5 times
[ 6033.040520] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 02:33:53 (1786170833)
[ 6046.176380] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6046.198447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 6046.219586] Lustre: Skipped 1 previous similar message
[ 6046.797651] Lustre: server umount lustre-MDT0000 complete
[ 6051.113439] LustreError: 5837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170851 with bad export cookie 16547204797136060062
[ 6051.118736] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6051.190098] Lustre: server umount lustre-OST0000 complete
[ 6055.848399] Lustre: server umount lustre-OST0001 complete
[ 6071.784120] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local
[ 6081.827093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6086.975674] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6090.026995] Lustre: 134461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6095.910141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6102.243206] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6107.175609] LustreError: 134828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6107.215308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:548 to 0x240000400:577)
[ 6110.101116] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6115.331847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:547 to 0x280000400:577)
[ 6116.045860] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6124.093646] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6128.219328] Lustre: 136391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6145.528846] Lustre: server umount lustre-MDT0000 complete
[ 6148.843213] LustreError: 134007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170949 with bad export cookie 16547204797136065585
[ 6148.856834] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6159.442250] Lustre: server umount lustre-OST0000 complete
[ 6161.503442] Lustre: 3315:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786170946/real 1786170946]  req@ffff98a1cdfb8700 x1872929669502976/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786170962 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
[ 6161.531116] Lustre: 3315:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 6162.577794] Lustre: server umount lustre-OST0001 complete
[ 6176.743303] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local
[ 6188.159211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6193.277887] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6196.735292] Lustre: 139093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6210.063140] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6212.457236] LustreError: 139459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6212.491940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:548 to 0x240000400:609)
[ 6217.696700] LustreError: 139459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6218.293232] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 6218.297616] Lustre: Skipped 1 previous similar message
[ 6223.350692] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:547 to 0x280000400:609)
[ 6224.404304] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6232.472254] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 6236.640861] Lustre: 141003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 6247.646883] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 02:37:27 (1786171047)
[ 6249.750822] Lustre: DEBUG MARKER: SKIP: sanity-quota test_23 Overwrite in place is not guaranteed to be space neutral on ZFS
[ 6251.632853] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 02:37:31 (1786171051)
[ 6311.926265] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 02:38:31 (1786171111)
[ 6358.054484] Lustre: DEBUG MARKER: Write...
[ 6360.267461] Lustre: DEBUG MARKER: Write out of block quota ...
[ 6414.328909] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 02:40:14 (1786171214)
[ 6421.224184] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 02:40:20 (1786171220)
[ 6431.556836] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 02:40:31 (1786171231)
[ 6440.595174] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 02:40:40 (1786171240)
[ 6448.926633] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 02:40:48 (1786171248)
[ 6516.847712] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 6675.586811] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 6871.848167] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 02:47:51 (1786171671)
[ 6935.372138] Lustre: DEBUG MARKER: Restart...
[ 6940.128718] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6940.129935] LustreError: 143333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6940.140290] Lustre: Skipped 3 previous similar messages
[ 6941.417557] Lustre: server umount lustre-MDT0000 complete
[ 6944.737490] LustreError: 138649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171745 with bad export cookie 16547204797136067118
[ 6944.748157] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6944.777192] LustreError: 138649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 6944.862702] Lustre: server umount lustre-OST0000 complete
[ 6948.805748] Lustre: server umount lustre-OST0001 complete
[ 6964.580201] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local
[ 6974.873074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6979.325620] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6981.613356] Lustre: 159990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 6988.393652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 6994.882809] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 6999.597753] LustreError: 161069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6999.612504] LustreError: 161069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message
[ 6999.615080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:618 to 0x240000400:641)
[ 7001.637282] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 7006.701904] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:618 to 0x280000400:641)
[ 7007.269483] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 7014.786091] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 7017.984401] Lustre: 161913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 7085.721864] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 02:51:24 (1786171884)
[ 7162.006933] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 02:52:41 (1786171961)
[ 8864.403188] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 03:21:04 (1786173664)
[ 8878.953595] Lustre: server umount lustre-MDT0000 complete
[ 8884.002765] LustreError: 159542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173684 with bad export cookie 16547204797136073614
[ 8884.014844] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8893.498548] Lustre: server umount lustre-OST0000 complete
[ 8907.584155] Lustre: server umount lustre-OST0001 complete
[ 8920.868410] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local
[ 8929.715383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8934.238628] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8936.882543] Lustre: 170399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17
[ 8943.461640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 8949.212765] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8954.743852] LustreError: 170770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8954.792971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:5643 to 0x240000400:5665)
[ 8957.400217] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 8962.548412] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5643 to 0x280000400:5665)
[ 8963.685588] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 8970.863867] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 8974.462682] Lustre: 172324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17
[ 9013.689972] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 03:23:33 (1786173813)
[ 9062.149824] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 03:24:22 (1786173862)
[ 9106.055282] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 03:25:06 (1786173906)
[ 9107.817357] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40c needs >= 2 MDTs
[ 9109.790259] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 03:25:09 (1786173909)
[ 9111.949149] Lustre: DEBUG MARKER: SKIP: sanity-quota test_40d needs >= 2 MDTs
[ 9113.370660] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 03:25:13 (1786173913)
[ 9211.510340] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 03:26:51 (1786174011)
[ 9262.196608] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 03:27:42 (1786174062)
[ 9352.775039] Lustre: DEBUG MARKER: == sanity-quota test 49a: lfs quota -a prints the quota usage for all quota IDs ========================================================== 03:29:12 (1786174152)
[ 9366.277760] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9366.282876] Lustre: Skipped 4 previous similar messages
[ 9367.310911] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9367.313088] Lustre: Skipped 59 previous similar messages
[ 9369.350344] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9369.352438] Lustre: Skipped 109 previous similar messages
[ 9373.389412] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9373.396557] Lustre: Skipped 189 previous similar messages
[ 9381.401648] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9381.403931] Lustre: Skipped 405 previous similar messages
[ 9397.403595] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9397.409746] Lustre: Skipped 725 previous similar messages
[ 9429.446997] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9429.450198] Lustre: Skipped 1689 previous similar messages
[ 9493.461114] Lustre: *** cfs_fail_loc=a09, val=0***
[ 9493.463312] Lustre: Skipped 3279 previous similar messages
[ 9901.432708] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9901.443384] Lustre: Skipped 1537 previous similar messages
[ 9901.506518] Lustre: *** cfs_fail_loc=a08, val=0***
[ 9901.515367] Lustre: Skipped 2 previous similar messages
[10041.314618] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[10041.315715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[10041.322214] Lustre: Skipped 1 previous similar message
[10041.328336] Lustre: Skipped 1 previous similar message
[10044.319583] Lustre: server umount lustre-MDT0000 complete
[10047.661198] LustreError: 169952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174848 with bad export cookie 16547204797137848548
[10047.672530] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[10048.185614] Lustre: server umount lustre-OST0000 complete
[10052.839792] Lustre: server umount lustre-OST0001 complete
[10060.226878] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid
[10069.147370] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local
[10078.737950]  vdc: vdc1 vdc9
[10088.141619]  vde: vde1 vde9
[10098.468876]  vdf: vdf1 vdf9
[10114.981988] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local
[10123.429215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000
[10123.692045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[10123.750989] Lustre: lustre-MDT0000: new disk, initializing
[10124.008398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[10124.068091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[10128.753155] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10134.143966] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params
[10140.397372] Lustre: lustre-OST0000: new disk, initializing
[10140.400781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[10140.407430] Lustre: Skipped 1 previous similar message
[10140.489882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[10142.417502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost
[10142.423380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost]
[10142.561411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400
[10145.804308] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10157.121462] Lustre: lustre-OST0001: new disk, initializing
[10157.129367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[10157.237621] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[10159.263049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost
[10159.268795] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost]
[10159.583392] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400
[10163.492339] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[10173.088843] Lustre: DEBUG MARKER: Using TIMEOUT=20
[10176.106300] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params
[10213.048830] Lustre: DEBUG MARKER: == sanity-quota test 49b: lfs quota -a --blocks has a delimiter ========================================================== 03:43:33 (1786175013)
[10219.942457] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 03:43:39 (1786175019)
[10267.690098] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 03:44:27 (1786175067)
[10328.955223] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 03:45:29 (1786175129)
[10358.736935] Lustre: DEBUG MARKER: rename directory return 255
[10404.577875] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 03:46:44 (1786175204)
[10438.815227] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 03:47:18 (1786175238)
[10482.175674] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 03:48:02 (1786175282)
[10565.622647] LustreError: 191153:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[10605.959723] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 03:50:06 (1786175406)
[10645.381056] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 03:50:45 (1786175445)
[10692.642413] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 03:51:32 (1786175492)
[10880.483947] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 03:54:40 (1786175680)
[10882.127960] Lustre: DEBUG MARKER: SKIP: sanity-quota test_59 ldiskfs only test
[10883.843894] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 03:54:43 (1786175683)
[10944.693259] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[10947.050987] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 03:55:46 (1786175746)
[10981.911850] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[10983.843646] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non-dir/files should succeed ========================================================== 03:56:23 (1786175783)
[11034.113947] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[11036.310472] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 03:57:16 (1786175836)
[11087.101536] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 03:58:06 (1786175886)
[11088.903629] Lustre: DEBUG MARKER: SKIP: sanity-quota test_67 ZFS grants some block space together with inode
[11091.192684] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 03:58:10 (1786175890)
[11118.013039] LustreError: 210463:0:(mgs_handler.c:1144:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[11162.812281] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 03:59:22 (1786175962)
[11191.503829] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[11193.269510] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[11243.114322] LustreError: 191153:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:10240 soft:0 granted:10240 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11337.994907] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 04:02:18 (1786176138)
[11392.286483] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 04:03:12 (1786176192)
[11424.469447] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 04:03:44 (1786176224)
[11425.793955] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71a ZFS grants some block space together with inode
[11427.682526] Lustre: DEBUG MARKER: == sanity-quota test 71b: Check SEL with quota pools ===== 04:03:47 (1786176227)
[11429.156409] Lustre: DEBUG MARKER: SKIP: sanity-quota test_71b ZFS grants some block space together with inode
[11430.802926] Lustre: DEBUG MARKER: == sanity-quota test 72: lfs quota --pool prints only pool's OSTs ========================================================== 04:03:50 (1786176230)
[11447.200388] Lustre: DEBUG MARKER: User quota (block hardlimit:50 MB)
[11462.862585] Lustre: DEBUG MARKER: Write...
[11465.165064] Lustre: DEBUG MARKER: Write out of block quota ...
[11518.731293] Lustre: DEBUG MARKER: == sanity-quota test 73a: default limits at OST Pool Quotas ========================================================== 04:05:18 (1786176318)
[11543.347185] Lustre: DEBUG MARKER: set to use default quota
[11544.927679] Lustre: DEBUG MARKER: set default quota
[11546.692518] Lustre: DEBUG MARKER: get default quota
[11553.054128] Lustre: DEBUG MARKER: Test not out of quota
[11556.788657] Lustre: DEBUG MARKER: Test out of quota
[11567.199139] Lustre: DEBUG MARKER: Increase default quota
[11595.518741] Lustre: DEBUG MARKER: Set quota to override default quota
[11595.561077] LustreError: 190306:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-qpool1 id:60000 enforced:1 hard:20480 soft:20480 granted:45063 time:1786781196 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[11607.339383] Lustre: DEBUG MARKER: Set to use default quota again
[11628.220589] Lustre: DEBUG MARKER: Cleanup
[11698.916927] Lustre: DEBUG MARKER: == sanity-quota test 73b: default OST Pool Quotas limit for new user ========================================================== 04:08:19 (1786176499)
[11722.974354] Lustre: DEBUG MARKER: set default quota for qpool1
[11724.340129] Lustre: DEBUG MARKER: Write from user that hasn't lqe
[11773.628739] Lustre: DEBUG MARKER: == sanity-quota test 74: check quota pools per user ====== 04:09:33 (1786176573)
[11853.755739] Lustre: DEBUG MARKER: == sanity-quota test 75: nodemap squashed root respects quota enforcement ========================================================== 04:10:54 (1786176654)
[11927.015835] Lustre: DEBUG MARKER: Write...
[11930.529530] Lustre: DEBUG MARKER: Write out of block quota ...
[12050.011736] Lustre: DEBUG MARKER: == sanity-quota test 76: project ID 4294967295 should be not allowed ========================================================== 04:14:09 (1786176849)
[12096.001962] Lustre: DEBUG MARKER: == sanity-quota test 77: lfs setquota should fail in Lustre mount with 'ro' ========================================================== 04:14:56 (1786176896)
[12103.107778] Lustre: DEBUG MARKER: == sanity-quota test 78A: Check fallocate increase quota usage ========================================================== 04:15:03 (1786176903)
[12105.026513] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78A need >= 2.13.57 and ldiskfs for fallocate
[12107.109520] Lustre: DEBUG MARKER: == sanity-quota test 78a: Check fallocate increase projectid usage ========================================================== 04:15:07 (1786176907)
[12108.669942] Lustre: DEBUG MARKER: SKIP: sanity-quota test_78a need >= 2.13.57 and ldiskfs for fallocate
[12110.505496] Lustre: DEBUG MARKER: == sanity-quota test 79: access to non-existed dt-pool/info doesn't cause a panic ========================================================== 04:15:10 (1786176910)
[12133.393949] Lustre: DEBUG MARKER: == sanity-quota test 80: check for EDQUOT after OST failover ========================================================== 04:15:33 (1786176933)
[12135.357425] Lustre: DEBUG MARKER: SKIP: sanity-quota test_80 ZFS grants some block space together with inode
[12137.051217] Lustre: DEBUG MARKER: == sanity-quota test 81: Race qmt_start_pool_recalc with qmt_pool_free ========================================================== 04:15:37 (1786176937)
[12154.702773] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[12163.027215] LustreError: 235606:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 sleeping for 10000ms
[12163.034812] LustreError: 235606:0:(qmt_pool.c:1407:qmt_pool_recalc()) Skipped 2 previous similar messages
[12167.138001] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[12167.142661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12167.151327] Lustre: Skipped 1 previous similar message
[12167.168105] Lustre: Skipped 1 previous similar message
[12168.497883] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping)
[12172.256080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[12172.265201] Lustre: Skipped 1 previous similar message
[12173.047586] LustreError: 235606:0:(qmt_pool.c:1407:qmt_pool_recalc()) cfs_fail_timeout id a07 awake
[12173.703074] LustreError: 187288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[12175.266645] Lustre: server umount lustre-MDT0000 complete
[12183.709280] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[12184.396138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[12184.499582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:27 to 0x240000400:65)
[12184.500505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:27 to 0x280000400:65)
[12188.369293] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[12196.835457] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo)
[12196.840095] Lustre: Skipped 1 previous similar message
[12229.787949] Lustre: DEBUG MARKER: == sanity-quota test 82: verify more than 8 qids for single operation ========================================================== 04:17:09 (1786177029)
[12264.729964] Lustre: DEBUG MARKER: == sanity-quota test 83: Setting default quota shouldn't affect grace time ========================================================== 04:17:44 (1786177064)
[12302.263420] Lustre: DEBUG MARKER: == sanity-quota test 84: Reset quota should fix the insane granted quota ========================================================== 04:18:22 (1786177102)
[12355.436630] Lustre: *** cfs_fail_loc=a08, val=0***
[12355.444736] Lustre: Skipped 14 previous similar messages
[12355.449381] Lustre: *** cfs_fail_loc=a08, val=0***
[12355.452796] Lustre: Skipped 7 previous similar messages
[12461.274583] Lustre: DEBUG MARKER: == sanity-quota test 85: do not hung at write with the least_qunit ========================================================== 04:21:01 (1786177261)
[12550.748531] Lustre: DEBUG MARKER: == sanity-quota test 86: Pre-acquired quota should be released if quota is over limit ========================================================== 04:22:30 (1786177350)
[12632.144239] LustreError: 237070:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786782232 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12816.174389] LustreError: 236656:0:(qmt_entry.c:558:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:md-0x0 id:60000 enforced:1 hard:2048 soft:2048 granted:65536 time:1786782416 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[12824.713198] Lustre: DEBUG MARKER: sanity-quota test_86: @@@@@@ FAIL: succeed to create files, expect failed