-----============= acceptance-small: replay-dual ============----- Wed May 8 17:13:34 EDT 2024 excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b Starting client oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre2 Started clients oleg125-client.virtnet: 192.168.201.125@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) oleg125-client.virtnet: executing check_config_client /mnt/lustre oleg125-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg125-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a992e800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b5ada000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a992e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5ada000.idle_timeout=debug disable quota as required oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 == replay-dual test 0a: expired recovery with lost client ========================================================== 17:13:43 (1715202823) Check file is LU482_FAILED=/tmp/replay-dual.lu482.jk2eEa UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 50 open/close in 0.23 seconds: 220.25 ops/second fail_loc=0x80000514 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1715202927 ; total 0 ; last 0) total: 50 unlinks in 0 seconds: inf unlinks/second PASS 0a (105s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 17:15:28 (1715202928) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre2 PASS 0b (92s) == replay-dual test 1: |X| simple create ================= 17:17:00 (1715203020) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (23s) == replay-dual test 2: |X| mkdir adir ==================== 17:17:23 (1715203043) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (40s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 17:18:03 (1715203083) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (26s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 17:18:29 (1715203109) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (32s) == replay-dual test 5: open, unlink |X| close ============ 17:19:01 (1715203141) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6844 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (23s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 17:19:24 (1715203164) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.6844 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.6844 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (23s) == replay-dual test 8: replay of resent request ========== 17:19:47 (1715203187) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (32s) == replay-dual test 9: resending a replayed create ======= 17:20:19 (1715203219) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (29s) == replay-dual test 10: resending a replayed unlink ====== 17:20:48 (1715203248) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (39s) == replay-dual test 11: both clients timeout during replay ========================================================== 17:21:27 (1715203287) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 13 sec fail_loc=0 PASS 11 (36s) == replay-dual test 12: open resend timeout ============== 17:22:03 (1715203323) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.6844 fail_loc=0x80000302 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (21s) == replay-dual test 13: close resend timeout ============= 17:22:24 (1715203344) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.6844 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (28s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 17:22:53 (1715203373) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.15 seconds: 167.76 ops/second total: 1 open/close in 0.01 seconds: 158.91 ops/second Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1715203469 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre2 PASS 15a (98s) == replay-dual test 15c: remove multiple OST orphans ===== 17:24:31 (1715203471) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre2 PASS 15c (108s) == replay-dual test 16: fail MDS during recovery (3571) == 17:26:19 (1715203579) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 9216 7529472 1% /mnt/lustre total: 25 open/close in 0.14 seconds: 182.76 ops/second total: 1 open/close in 0.01 seconds: 181.23 ops/second Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1715203721 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre2 PASS 16 (144s) == replay-dual test 17: fail OST during recovery (3571) == 17:28:43 (1715203723) total: 25 open/close in 0.12 seconds: 211.31 ops/second total: 1 open/close in 0.01 seconds: 135.08 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing ost1 on oleg125-server Stopping /mnt/lustre-ost1 (opts:) on oleg125-server reboot facets: ost1 Failover ost1 to oleg125-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg125-server Stopping /mnt/lustre-ost1 (opts:) on oleg125-server reboot facets: ost1 Failover ost1 to oleg125-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1715203839 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre2 PASS 17 (118s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 17:30:41 (1715203841) debug=+dlmtrace using seed 4015203323 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.201.125@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a99f6000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9bf7800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800a99f6000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800a9bf7800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800a99f6000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800a9bf7800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.201.125@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a99f6000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9bf7800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800a99f6000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800a9bf7800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800a99f6000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800a9bf7800.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (44s) == replay-dual test 19: resend of open request =========== 17:31:25 (1715203885) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1715203963.34 total 76.02 last 0.00) total: 1 open/close in 76.02 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (102s) == replay-dual test 20: recovery time is not increasing == 17:33:07 (1715203987) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre2 PASS 20 (310s) == replay-dual test 21a: commit on sharing =============== 17:38:17 (1715204297) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 PASS 21a (155s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 17:40:52 (1715204452) SKIP: replay-dual test_22a needs >= 2 MDTs SKIP 22a (1s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 17:40:53 (1715204453) SKIP: replay-dual test_22b needs >= 2 MDTs SKIP 22b (1s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 17:40:54 (1715204454) SKIP: replay-dual test_22c needs >= 2 MDTs SKIP 22c (0s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 17:40:54 (1715204454) SKIP: replay-dual test_22d needs >= 2 MDTs SKIP 22d (1s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 17:40:55 (1715204455) SKIP: replay-dual test_23a needs >= 2 MDTs SKIP 23a (1s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 17:40:56 (1715204456) SKIP: replay-dual test_23b needs >= 2 MDTs SKIP 23b (1s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 17:40:57 (1715204457) SKIP: replay-dual test_23c needs >= 2 MDTs SKIP 23c (0s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 17:40:57 (1715204457) SKIP: replay-dual test_23d needs >= 2 MDTs SKIP 23d (1s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 17:40:58 (1715204458) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (78s) == replay-dual test 25: replay|resend ==================== 17:42:16 (1715204536) 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00302069 s, 169 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg125-server Stopping /mnt/lustre-ost1 (opts:) on oleg125-server reboot facets: ost1 Failover ost1 to oleg125-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5977: 14213 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (20s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 17:42:36 (1715204556) Starting client oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre Started clients oleg125-client.virtnet: 192.168.201.125@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 15799 Started dbench loop with 15800 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed May 8 17:42:38 EDT 2024 waiting for dbench pid 15825 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 271 10.18 MB/sec warmup 1 sec latency 38.154 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3676160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 29696 3650560 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 32768 7326720 1% /mnt/lustre 1 530 9.19 MB/sec warmup 2 sec latency 17.824 ms 1 849 7.19 MB/sec warmup 3 sec latency 57.730 ms 1 1163 5.75 MB/sec warmup 4 sec latency 57.067 ms test_26 fail mds1 1 times Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server 1 1516 5.21 MB/sec warmup 5 sec latency 119.155 ms reboot facets: mds1 1 1516 4.34 MB/sec warmup 6 sec latency 1119.380 ms 1 1516 3.72 MB/sec warmup 7 sec latency 2119.597 ms 1 1516 3.26 MB/sec warmup 8 sec latency 3119.835 ms 1 1516 2.90 MB/sec warmup 9 sec latency 4120.101 ms 1 1516 2.61 MB/sec warmup 10 sec latency 5120.296 ms 1 1516 2.37 MB/sec warmup 11 sec latency 6120.477 ms 1 1516 2.17 MB/sec warmup 12 sec latency 7120.672 ms 1 1516 2.01 MB/sec warmup 13 sec latency 8120.818 ms 1 1516 1.86 MB/sec warmup 14 sec latency 9121.041 ms 1 1516 1.74 MB/sec warmup 15 sec latency 10121.224 ms Failover mds1 to oleg125-server mount facets: mds1 1 1516 1.63 MB/sec warmup 16 sec latency 11121.379 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1516 1.53 MB/sec warmup 17 sec latency 12121.554 ms 1 1516 1.45 MB/sec warmup 18 sec latency 13121.777 ms 1 1516 1.37 MB/sec warmup 19 sec latency 14122.006 ms oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 1 1516 0.00 MB/sec execute 1 sec latency 16122.320 ms Started lustre-MDT0000 1 1516 0.00 MB/sec execute 2 sec latency 17122.451 ms 1 1516 0.00 MB/sec execute 3 sec latency 18122.632 ms 1 1516 0.00 MB/sec execute 4 sec latency 19122.824 ms 1 1516 0.00 MB/sec execute 5 sec latency 20123.025 ms 1 1516 0.00 MB/sec execute 6 sec latency 21123.198 ms 1 1516 0.00 MB/sec execute 7 sec latency 22123.365 ms 1 1516 0.00 MB/sec execute 8 sec latency 23123.556 ms 1 1516 0.00 MB/sec execute 9 sec latency 24123.781 ms 1 1516 0.00 MB/sec execute 10 sec latency 25123.963 ms 1 1516 0.00 MB/sec execute 11 sec latency 26124.152 ms 1 1516 0.00 MB/sec execute 12 sec latency 27124.339 ms 1 1516 0.00 MB/sec execute 13 sec latency 28124.514 ms 1 1516 0.00 MB/sec execute 14 sec latency 29124.681 ms 1 1556 0.00 MB/sec execute 15 sec latency 30010.461 ms oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1925 0.02 MB/sec execute 16 sec latency 42.031 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2345 0.11 MB/sec execute 17 sec latency 31.829 ms 1 2720 0.33 MB/sec execute 18 sec latency 42.340 ms 1 3186 0.43 MB/sec execute 19 sec latency 54.949 ms 1 3647 0.63 MB/sec execute 20 sec latency 59.525 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4992 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3666944 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3676160 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 59392 7343104 1% /mnt/lustre 1 3939 0.67 MB/sec execute 21 sec latency 18.296 ms 1 4208 0.64 MB/sec execute 22 sec latency 15.719 ms 1 4421 0.64 MB/sec execute 23 sec latency 84.303 ms test_26 fail mds1 2 times Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server 1 4701 0.67 MB/sec execute 24 sec latency 157.242 ms reboot facets: mds1 1 4701 0.64 MB/sec execute 25 sec latency 1157.454 ms 1 4701 0.62 MB/sec execute 26 sec latency 2157.672 ms 1 4701 0.59 MB/sec execute 27 sec latency 3157.908 ms 1 4701 0.57 MB/sec execute 28 sec latency 4158.116 ms 1 4701 0.55 MB/sec execute 29 sec latency 5158.308 ms 1 4701 0.53 MB/sec execute 30 sec latency 6158.494 ms 1 4701 0.52 MB/sec execute 31 sec latency 7158.698 ms 1 4701 0.50 MB/sec execute 32 sec latency 8158.899 ms 1 4701 0.49 MB/sec execute 33 sec latency 9159.166 ms 1 4701 0.47 MB/sec execute 34 sec latency 10159.359 ms Failover mds1 to oleg125-server mount facets: mds1 1 4701 0.46 MB/sec execute 35 sec latency 11159.527 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4701 0.45 MB/sec execute 36 sec latency 12159.688 ms 1 4701 0.43 MB/sec execute 37 sec latency 13159.825 ms 1 4701 0.42 MB/sec execute 38 sec latency 14159.939 ms oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 1 4701 0.41 MB/sec execute 39 sec latency 15160.128 ms pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4701 0.40 MB/sec execute 40 sec latency 16160.242 ms 1 4756 0.39 MB/sec execute 41 sec latency 17004.707 ms oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5151 0.46 MB/sec execute 42 sec latency 43.833 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5468 0.45 MB/sec execute 43 sec latency 38.769 ms 1 5913 0.48 MB/sec execute 44 sec latency 45.364 ms 1 6335 0.56 MB/sec execute 45 sec latency 40.832 ms 1 6933 0.62 MB/sec execute 46 sec latency 33.050 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 6912 2201728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3446784 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 44032 3403776 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 64512 6850560 1% /mnt/lustre 1 7368 0.70 MB/sec execute 47 sec latency 16.466 ms 1 7637 0.69 MB/sec execute 48 sec latency 15.864 ms test_26 fail mds1 3 times 1 7909 0.69 MB/sec execute 49 sec latency 80.163 ms Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server 1 8111 0.68 MB/sec execute 50 sec latency 224.822 ms reboot facets: mds1 1 8111 0.67 MB/sec execute 51 sec latency 1224.984 ms 1 8111 0.66 MB/sec execute 52 sec latency 2225.168 ms 1 8111 0.65 MB/sec execute 53 sec latency 3225.385 ms 1 8111 0.63 MB/sec execute 54 sec latency 4225.650 ms 1 8111 0.62 MB/sec execute 55 sec latency 5225.844 ms 1 8111 0.61 MB/sec execute 56 sec latency 6226.019 ms 1 8111 0.60 MB/sec execute 57 sec latency 7226.168 ms 1 8111 0.59 MB/sec execute 58 sec latency 8226.376 ms 1 8111 0.58 MB/sec execute 59 sec latency 9226.639 ms 1 8111 0.57 MB/sec execute 60 sec latency 10226.834 ms Failover mds1 to oleg125-server mount facets: mds1 1 8111 0.56 MB/sec execute 61 sec latency 11227.030 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 8111 0.55 MB/sec execute 62 sec latency 12227.251 ms 1 8111 0.54 MB/sec execute 63 sec latency 13227.556 ms 1 8111 0.53 MB/sec execute 64 sec latency 14227.784 ms 1 8111 0.53 MB/sec execute 65 sec latency 15228.021 ms 1 8111 0.52 MB/sec execute 66 sec latency 16228.171 ms oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 1 8111 0.51 MB/sec execute 67 sec latency 17228.398 ms pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 1 8111 0.50 MB/sec execute 68 sec latency 18228.625 ms 1 8111 0.50 MB/sec execute 69 sec latency 19228.917 ms 1 8111 0.49 MB/sec execute 70 sec latency 20229.156 ms 1 8111 0.48 MB/sec execute 71 sec latency 21229.404 ms 1 8111 0.48 MB/sec execute 72 sec latency 22229.560 ms 1 8111 0.47 MB/sec execute 73 sec latency 23229.750 ms 1 8111 0.46 MB/sec execute 74 sec latency 24229.940 ms 1 8111 0.46 MB/sec execute 75 sec latency 25230.125 ms 1 8111 0.45 MB/sec execute 76 sec latency 26230.340 ms 1 8111 0.44 MB/sec execute 77 sec latency 27230.545 ms 1 8111 0.44 MB/sec execute 78 sec latency 28230.775 ms 1 8111 0.43 MB/sec execute 79 sec latency 29230.951 ms 1 8111 0.43 MB/sec execute 80 sec latency 30231.086 ms 1 8405 0.44 MB/sec execute 81 sec latency 30456.357 ms oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8763 0.47 MB/sec execute 82 sec latency 53.341 ms 1 9142 0.47 MB/sec execute 83 sec latency 49.675 ms 1 9541 0.48 MB/sec execute 84 sec latency 40.262 ms 1 10089 0.54 MB/sec execute 85 sec latency 40.935 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 8960 2199680 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3655680 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 47104 3643392 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 7299072 1% /mnt/lustre 1 10675 0.59 MB/sec execute 86 sec latency 38.744 ms 1 10995 0.60 MB/sec execute 87 sec latency 13.094 ms 1 11316 0.59 MB/sec execute 88 sec latency 14.524 ms test_26 fail mds1 4 times Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server 1 11551 0.60 MB/sec execute 89 sec latency 56.599 ms reboot facets: mds1 1 11594 0.59 MB/sec execute 90 sec latency 827.626 ms 1 11594 0.58 MB/sec execute 91 sec latency 1827.805 ms 1 11594 0.58 MB/sec execute 92 sec latency 2828.000 ms 1 11594 0.57 MB/sec execute 93 sec latency 3828.178 ms 1 11594 0.56 MB/sec execute 94 sec latency 4828.362 ms 1 11594 0.56 MB/sec execute 95 sec latency 5828.561 ms 1 11594 0.55 MB/sec execute 96 sec latency 6828.784 ms 1 11594 0.55 MB/sec execute 97 sec latency 7828.994 ms 1 11594 0.54 MB/sec execute 98 sec latency 8829.178 ms 1 11594 0.54 MB/sec execute 99 sec latency 9829.375 ms Failover mds1 to oleg125-server 1 cleanup 100 sec mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 1 cleanup 106 sec 0 cleanup 107 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1747 34.727 30456.342 Close 1273 1.412 5.164 Rename 71 249.692 17004.688 Unlink 361 87.080 30010.448 Qpathinfo 1601 1.966 22.576 Qfileinfo 275 0.358 2.313 Qfsinfo 289 0.374 2.086 Sfileinfo 131 4.862 9.240 Find 614 1.062 12.557 WriteX 856 1.719 8.949 ReadX 2723 0.071 26.305 LockX 6 1.521 2.234 UnlockX 6 1.495 1.979 Flush 126 25.591 84.287 Throughput 0.535892 MB/sec 1 clients 1 procs max_latency=30456.357 ms stopping dbench on /mnt/lustre at Wed May 8 17:44:45 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Wed May 8 17:44:46 EDT 2024 waiting for dbench pid 19936 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 289 10.70 MB/sec warmup 1 sec latency 20.297 ms 1 639 10.52 MB/sec warmup 2 sec latency 22.346 ms 1 873 7.22 MB/sec warmup 3 sec latency 88.321 ms 1 1214 5.76 MB/sec warmup 4 sec latency 31.492 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10496 2198144 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3290112 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 26624 3301376 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 6591488 1% /mnt/lustre 1 1619 5.23 MB/sec warmup 5 sec latency 66.096 ms 1 2004 4.42 MB/sec warmup 6 sec latency 34.729 ms 1 2416 4.01 MB/sec warmup 7 sec latency 42.048 ms test_26 fail mds1 5 times Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server 1 2934 4.19 MB/sec warmup 8 sec latency 43.164 ms reboot facets: mds1 1 2934 3.72 MB/sec warmup 9 sec latency 1043.378 ms 1 2934 3.35 MB/sec warmup 10 sec latency 2043.565 ms 1 2934 3.04 MB/sec warmup 11 sec latency 3043.759 ms 1 2934 2.79 MB/sec warmup 12 sec latency 4044.009 ms 1 2934 2.58 MB/sec warmup 13 sec latency 5044.208 ms 1 2934 2.39 MB/sec warmup 14 sec latency 6044.403 ms 1 2934 2.23 MB/sec warmup 15 sec latency 7044.611 ms 1 2934 2.09 MB/sec warmup 16 sec latency 8044.804 ms 1 2934 1.97 MB/sec warmup 17 sec latency 9045.040 ms 1 2934 1.86 MB/sec warmup 18 sec latency 10045.231 ms Failover mds1 to oleg125-server mount facets: mds1 1 2934 1.76 MB/sec warmup 19 sec latency 11045.341 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2934 0.00 MB/sec execute 1 sec latency 13045.606 ms 1 2934 0.00 MB/sec execute 2 sec latency 14045.797 ms 1 2934 0.00 MB/sec execute 3 sec latency 15045.987 ms 1 2934 0.00 MB/sec execute 4 sec latency 16046.185 ms 1 2934 0.00 MB/sec execute 5 sec latency 17046.318 ms oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 1 2934 0.00 MB/sec execute 6 sec latency 18046.462 ms Started lustre-MDT0000 1 2934 0.00 MB/sec execute 7 sec latency 19046.633 ms 1 2934 0.00 MB/sec execute 8 sec latency 20046.885 ms 1 2934 0.00 MB/sec execute 9 sec latency 21047.063 ms 1 2934 0.00 MB/sec execute 10 sec latency 22047.255 ms 1 2934 0.00 MB/sec execute 11 sec latency 23047.461 ms 1 2934 0.00 MB/sec execute 12 sec latency 24047.662 ms 1 2934 0.00 MB/sec execute 13 sec latency 25047.867 ms 1 2934 0.00 MB/sec execute 14 sec latency 26048.056 ms 1 2934 0.00 MB/sec execute 15 sec latency 27048.184 ms 1 2934 0.00 MB/sec execute 16 sec latency 28048.390 ms 1 2934 0.00 MB/sec execute 17 sec latency 29048.526 ms 1 3170 0.04 MB/sec execute 18 sec latency 29712.021 ms oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3653 0.27 MB/sec execute 19 sec latency 40.611 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3953 0.33 MB/sec execute 20 sec latency 14.888 ms 1 4238 0.32 MB/sec execute 21 sec latency 118.559 ms 1 4486 0.34 MB/sec execute 22 sec latency 65.609 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10496 2198144 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 54272 3694592 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3715072 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 95232 7409664 2% /mnt/lustre 1 4851 0.38 MB/sec execute 23 sec latency 28.604 ms 1 5227 0.49 MB/sec execute 24 sec latency 63.695 ms 1 5626 0.49 MB/sec execute 25 sec latency 31.831 ms test_26 fail mds1 6 times 1 6015 0.53 MB/sec execute 26 sec latency 44.375 ms Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server 1 6329 0.65 MB/sec execute 27 sec latency 440.542 ms reboot facets: mds1 1 6329 0.63 MB/sec execute 28 sec latency 1440.754 ms 1 6329 0.61 MB/sec execute 29 sec latency 2440.977 ms 1 6329 0.59 MB/sec execute 30 sec latency 3441.237 ms 1 6329 0.57 MB/sec execute 31 sec latency 4441.520 ms 1 6329 0.55 MB/sec execute 32 sec latency 5441.793 ms 1 6329 0.53 MB/sec execute 33 sec latency 6442.010 ms 1 6329 0.52 MB/sec execute 34 sec latency 7442.305 ms 1 6329 0.50 MB/sec execute 35 sec latency 8442.560 ms 1 6329 0.49 MB/sec execute 36 sec latency 9442.831 ms 1 6329 0.48 MB/sec execute 37 sec latency 10443.115 ms Failover mds1 to oleg125-server mount facets: mds1 1 6329 0.46 MB/sec execute 38 sec latency 11443.441 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 6329 0.45 MB/sec execute 39 sec latency 12443.695 ms 1 6329 0.44 MB/sec execute 40 sec latency 13443.931 ms 1 6329 0.43 MB/sec execute 41 sec latency 14444.091 ms 1 6329 0.42 MB/sec execute 42 sec latency 15444.263 ms 1 6329 0.41 MB/sec execute 43 sec latency 16444.372 ms oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 1 6329 0.40 MB/sec execute 44 sec latency 17444.503 ms Started lustre-MDT0000 1 6429 0.40 MB/sec execute 45 sec latency 18332.159 ms 1 6922 0.46 MB/sec execute 46 sec latency 41.217 ms oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7360 0.54 MB/sec execute 47 sec latency 14.928 ms 1 7623 0.54 MB/sec execute 48 sec latency 14.388 ms 1 7902 0.53 MB/sec execute 49 sec latency 83.775 ms 1 8190 0.55 MB/sec execute 50 sec latency 40.358 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 7296 2201216 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 49152 3706880 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3727360 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 7434240 1% /mnt/lustre 1 8590 0.60 MB/sec execute 51 sec latency 50.429 ms 1 8887 0.60 MB/sec execute 52 sec latency 51.082 ms 1 9303 0.59 MB/sec execute 53 sec latency 50.839 ms test_26 fail mds1 7 times Failing mds1 on oleg125-server 1 9696 0.62 MB/sec execute 54 sec latency 45.468 ms Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 1 9993 0.68 MB/sec execute 55 sec latency 640.611 ms 1 9993 0.67 MB/sec execute 56 sec latency 1640.777 ms 1 9993 0.66 MB/sec execute 57 sec latency 2640.971 ms 1 9993 0.65 MB/sec execute 58 sec latency 3641.123 ms 1 9993 0.64 MB/sec execute 59 sec latency 4641.267 ms 1 9993 0.63 MB/sec execute 60 sec latency 5641.454 ms 1 9993 0.62 MB/sec execute 61 sec latency 6641.627 ms 1 9993 0.61 MB/sec execute 62 sec latency 7641.773 ms 1 9993 0.60 MB/sec execute 63 sec latency 8641.966 ms 1 9993 0.59 MB/sec execute 64 sec latency 9642.135 ms 1 9993 0.58 MB/sec execute 65 sec latency 10642.241 ms Failover mds1 to oleg125-server mount facets: mds1 1 9993 0.57 MB/sec execute 66 sec latency 11642.330 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 9993 0.56 MB/sec execute 67 sec latency 12642.631 ms 1 9993 0.55 MB/sec execute 68 sec latency 13642.808 ms 1 9993 0.54 MB/sec execute 69 sec latency 14643.204 ms 1 9993 0.54 MB/sec execute 70 sec latency 15643.399 ms 1 9993 0.53 MB/sec execute 71 sec latency 16643.609 ms oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 1 9993 0.52 MB/sec execute 72 sec latency 17643.792 ms pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 1 9993 0.51 MB/sec execute 73 sec latency 18643.993 ms 1 9993 0.51 MB/sec execute 74 sec latency 19644.174 ms 1 9993 0.50 MB/sec execute 75 sec latency 20644.719 ms 1 9993 0.49 MB/sec execute 76 sec latency 21644.877 ms 1 9993 0.49 MB/sec execute 77 sec latency 22645.029 ms 1 9993 0.48 MB/sec execute 78 sec latency 23645.166 ms 1 9993 0.48 MB/sec execute 79 sec latency 24645.330 ms 1 9993 0.47 MB/sec execute 80 sec latency 25645.490 ms 1 9993 0.46 MB/sec execute 81 sec latency 26645.631 ms 1 9993 0.46 MB/sec execute 82 sec latency 27645.778 ms 1 9993 0.45 MB/sec execute 83 sec latency 28645.899 ms 1 9993 0.45 MB/sec execute 84 sec latency 29646.064 ms 1 10396 0.47 MB/sec execute 85 sec latency 29749.216 ms oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 10811 0.51 MB/sec execute 86 sec latency 28.286 ms 1 11095 0.52 MB/sec execute 87 sec latency 15.675 ms 1 11393 0.51 MB/sec execute 88 sec latency 82.458 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Wed May 8 17:46:35 EDT 2024 with return code 0 19936 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 19936 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (240s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 17:46:36 (1715204796) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00238137 s, 1.7 MB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg125-server Stopping /mnt/lustre-ost1 (opts:) on oleg125-server reboot facets: ost1 Failover ost1 to oleg125-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00455499 s, 899 kB/s oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (26s) == replay-dual test 29: replay vs update with the same xid ========================================================== 17:47:02 (1715204822) SKIP: replay-dual test_29 needs >= 2 MDTs SKIP 29 (1s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 17:47:03 (1715204823) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.00692438 s, 5.9 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.00599457 s, 6.8 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg125-server Stopping /mnt/lustre-mds1 (opts:) on oleg125-server reboot facets: mds1 Failover mds1 to oleg125-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 160+0 records in 160+0 records out 81920 bytes (82 kB) copied, 31.8932 s, 2.6 kB/s oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (36s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 17:47:39 (1715204859) Failing ost1 on oleg125-server Stopping /mnt/lustre-ost1 (opts:) on oleg125-server reboot facets: ost1 Creating to objid 3297 on ost lustre-OST0000... total: 32 open/close in 0.18 seconds: 177.54 ops/second at_max=0 fail_loc=0x80001420 Failover ost1 to oleg125-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL IDLE state after 0 sec at_max=600 PASS 31 (26s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 17:48:05 (1715204885) SKIP: replay-dual test_32 needs >= 2 MDTs SKIP 32 (0s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 17:48:06 (1715204886) SKIP: replay-dual test_33 ldiskfs only test SKIP 33 (1s) == replay-dual test complete, duration 2072 sec ========== 17:48:06 (1715204886) Stopping clients: oleg125-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg125-client.virtnet /mnt/lustre2 opts: