-----============= acceptance-small: replay-dual ============----- Fri Jan 17 11:16:32 EST 2025 excepting tests: 14b 21b skipping tests SLOW=no: 21b Starting client oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 Started clients oleg201-client.virtnet: 192.168.202.101@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) oleg201-client.virtnet: executing check_config_client /mnt/lustre oleg201-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg201-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a974f800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800aaee5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a974f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800aaee5800.idle_timeout=debug disable quota as required oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 osd-ldiskfs.track_declares_assert=1 == replay-dual test 0a: expired recovery with lost client ========================================================== 11:16:43 (1737130603) Check file is LU482_FAILED=/tmp/replay-dual.lu482.kQlEeN UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 50 open/close in 0.29 seconds: 172.46 ops/second fail_loc=0x80000514 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1737130702 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second PASS 0a (101s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 11:18:24 (1737130704) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 PASS 0b (127s) == replay-dual test 1: |X| simple create ================= 11:20:31 (1737130831) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg201-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (28s) == replay-dual test 2: |X| mkdir adir ==================== 11:20:59 (1737130859) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (28s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 11:21:27 (1737130887) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (29s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 11:21:56 (1737130916) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1692 1285996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (31s) == replay-dual test 5: open, unlink |X| close ============ 11:22:28 (1737130948) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7448 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1920 1285768 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1764 1285924 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (39s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 11:23:07 (1737130987) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7448 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7448 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (33s) == replay-dual test 8: replay of resent request ========== 11:23:40 (1737131020) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (38s) == replay-dual test 9: resending a replayed create ======= 11:24:18 (1737131058) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (46s) == replay-dual test 10: resending a replayed unlink ====== 11:25:04 (1737131104) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (45s) == replay-dual test 11: both clients timeout during replay ========================================================== 11:25:49 (1737131149) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec fail_loc=0 PASS 11 (40s) == replay-dual test 12: open resend timeout ============== 11:26:29 (1737131189) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7448 fail_loc=0x80000302 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (28s) == replay-dual test 13: close resend timeout ============= 11:26:57 (1737131217) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7448 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (29s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 11:27:27 (1737131247) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.23 seconds: 110.33 ops/second total: 1 open/close in 0.01 seconds: 97.21 ops/second Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1737131338 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 PASS 15a (92s) == replay-dual test 15c: remove multiple OST orphans ===== 11:28:59 (1737131339) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 PASS 15c (107s) == replay-dual test 16: fail MDS during recovery (3571) == 11:30:46 (1737131446) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.14 seconds: 173.78 ops/second total: 1 open/close in 0.01 seconds: 183.16 ops/second Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1737131572 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 PASS 16 (127s) == replay-dual test 17: fail OST during recovery (3571) == 11:32:53 (1737131573) total: 25 open/close in 0.15 seconds: 171.92 ops/second total: 1 open/close in 0.01 seconds: 86.09 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1728 1285960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing ost1 on oleg201-server Stopping /mnt/lustre-ost1 (opts:) on oleg201-server reboot facets: ost1 Failover ost1 to oleg201-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg201-server Stopping /mnt/lustre-ost1 (opts:) on oleg201-server reboot facets: ost1 Failover ost1 to oleg201-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1737131694 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 PASS 17 (122s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 11:34:55 (1737131695) debug=+dlmtrace using seed 3186792299 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.202.101@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8ccf800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9cfb800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8ccf800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a9cfb800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800a8ccf800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800a9cfb800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800a8ccf800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800a9cfb800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.202.101@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8ccf800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9cfb800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8ccf800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a9cfb800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800a8ccf800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800a9cfb800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800a8ccf800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800a9cfb800.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (45s) == replay-dual test 19: resend of open request =========== 11:35:40 (1737131740) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1732 1285956 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1737131818.80 total 76.02 last 0.00) total: 1 open/close in 76.02 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (104s) == replay-dual test 20: recovery time is not increasing == 11:37:24 (1737131844) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1732 1285956 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1732 1285956 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 PASS 20 (317s) == replay-dual test 21a: commit on sharing =============== 11:42:41 (1737132161) mdt.lustre-MDT0000.commit_on_sharing=1 mdt.lustre-MDT0001.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 mdt.lustre-MDT0001.commit_on_sharing=0 PASS 21a (158s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 11:45:20 (1737132320) fail_loc=0x119 Failing mds2 on oleg201-server Stopping /mnt/lustre-mds2 (opts:) on oleg201-server reboot facets: mds2 Failover mds2 to oleg201-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1896 1285792 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1756 1285932 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 104.38 ops/second total: 2 open/close in 0.01 seconds: 207.49 ops/second Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22a (52s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 11:46:12 (1737132372) fail_loc=0x119 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server Failing mds2 on oleg201-server Stopping /mnt/lustre-mds2 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg201-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1752 1285936 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 97.86 ops/second total: 2 open/close in 0.01 seconds: 218.85 ops/second Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22b (84s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 11:47:36 (1737132456) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1752 1285936 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 109.34 ops/second total: 2 open/close in 0.01 seconds: 220.05 ops/second Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22c (56s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 11:48:32 (1737132512) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server Failing mds2 on oleg201-server Stopping /mnt/lustre-mds2 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg201-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1716 1285972 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 97.09 ops/second total: 2 open/close in 0.01 seconds: 210.04 ops/second Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22d (89s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 11:50:01 (1737132601) fail_loc=0x119 fail_loc=0 Failing mds2 on oleg201-server Stopping /mnt/lustre-mds2 (opts:) on oleg201-server reboot facets: mds2 Failover mds2 to oleg201-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 61.82 ops/second Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23a (56s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 11:50:57 (1737132657) fail_loc=0x119 fail_loc=0 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server Failing mds2 on oleg201-server Stopping /mnt/lustre-mds2 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg201-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.01 seconds: 141.02 ops/second Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23b (74s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 11:52:11 (1737132731) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 112.26 ops/second Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23c (56s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 11:53:07 (1737132787) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server Failing mds2 on oleg201-server Stopping /mnt/lustre-mds2 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg201-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1712 1285976 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.01 seconds: 144.06 ops/second Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23d (88s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 11:54:35 (1737132875) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (77s) == replay-dual test 25: replay|resend ==================== 11:55:52 (1737132952) 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00209722 s, 244 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg201-server Stopping /mnt/lustre-ost1 (opts:) on oleg201-server reboot facets: ost1 Failover ost1 to oleg201-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5971: 4802 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (21s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 11:56:13 (1737132973) Starting client oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre Started clients oleg201-client.virtnet: 192.168.202.101@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 6403 Started dbench loop with 6404 striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jan 17 11:56:15 EST 2025 waiting for dbench pid 6439 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 279 10.44 MB/sec warmup 1 sec latency 19.937 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2112 1285576 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25984 3556032 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27372 7161664 1% /mnt/lustre 1 542 9.55 MB/sec warmup 2 sec latency 16.308 ms 1 914 7.26 MB/sec warmup 3 sec latency 15.660 ms 1 1376 6.50 MB/sec warmup 4 sec latency 14.423 ms test_26 fail mds1 1 times 1 1775 5.26 MB/sec warmup 5 sec latency 12.987 ms Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server 1 2096 4.44 MB/sec warmup 6 sec latency 237.089 ms reboot facets: mds1 1 2096 3.81 MB/sec warmup 7 sec latency 1237.256 ms 1 2096 3.33 MB/sec warmup 8 sec latency 2237.414 ms 1 2096 2.96 MB/sec warmup 9 sec latency 3237.571 ms 1 2096 2.67 MB/sec warmup 10 sec latency 4237.744 ms 1 2096 2.42 MB/sec warmup 11 sec latency 5237.908 ms 1 2096 2.22 MB/sec warmup 12 sec latency 6238.079 ms 1 2096 2.05 MB/sec warmup 13 sec latency 7238.212 ms 1 2096 1.90 MB/sec warmup 14 sec latency 8238.343 ms 1 2096 1.78 MB/sec warmup 15 sec latency 9238.509 ms 1 2096 1.67 MB/sec warmup 16 sec latency 10238.694 ms Failover mds1 to oleg201-server mount facets: mds1 1 2096 1.57 MB/sec warmup 17 sec latency 11238.851 ms 1 2096 1.48 MB/sec warmup 18 sec latency 12238.982 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 2096 1.40 MB/sec warmup 19 sec latency 13239.121 ms oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 1 2096 0.00 MB/sec execute 1 sec latency 15239.426 ms pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 1 2096 0.00 MB/sec execute 2 sec latency 16239.629 ms 1 2096 0.00 MB/sec execute 3 sec latency 17239.884 ms 1 2096 0.00 MB/sec execute 4 sec latency 18240.077 ms 1 2096 0.00 MB/sec execute 5 sec latency 19240.235 ms 1 2296 0.18 MB/sec execute 6 sec latency 19809.889 ms oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2754 0.76 MB/sec execute 7 sec latency 18.821 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3367 1.10 MB/sec execute 8 sec latency 15.015 ms 1 3808 1.46 MB/sec execute 9 sec latency 14.059 ms 1 4065 1.34 MB/sec execute 10 sec latency 13.877 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3056 1284632 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2588 1285100 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 29772 3569108 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5608 3591164 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 35380 7160272 1% /mnt/lustre 1 4388 1.27 MB/sec execute 11 sec latency 23.142 ms 1 4729 1.29 MB/sec execute 12 sec latency 18.150 ms 1 5143 1.43 MB/sec execute 13 sec latency 14.206 ms 1 5534 1.35 MB/sec execute 14 sec latency 15.019 ms test_26 fail mds2 2 times 1 5954 1.36 MB/sec execute 15 sec latency 37.399 ms Failing mds2 on oleg201-server Stopping /mnt/lustre-mds2 (opts:) on oleg201-server 1 6471 1.62 MB/sec execute 16 sec latency 16.433 ms reboot facets: mds2 1 7101 1.82 MB/sec execute 17 sec latency 14.534 ms 1 7429 1.79 MB/sec execute 18 sec latency 14.889 ms 1 7627 1.71 MB/sec execute 19 sec latency 257.523 ms 1 7627 1.63 MB/sec execute 20 sec latency 1257.653 ms 1 7627 1.55 MB/sec execute 21 sec latency 2257.815 ms 1 7627 1.48 MB/sec execute 22 sec latency 3257.903 ms 1 7627 1.42 MB/sec execute 23 sec latency 4258.026 ms 1 7627 1.36 MB/sec execute 24 sec latency 5258.187 ms 1 7627 1.30 MB/sec execute 25 sec latency 6258.316 ms 1 7627 1.25 MB/sec execute 26 sec latency 7258.450 ms Failover mds2 to oleg201-server mount facets: mds2 1 7627 1.21 MB/sec execute 27 sec latency 8258.556 ms 1 7627 1.16 MB/sec execute 28 sec latency 9258.670 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 7627 1.12 MB/sec execute 29 sec latency 10258.796 ms oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 1 7627 1.08 MB/sec execute 30 sec latency 11258.895 ms pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 1 7627 1.05 MB/sec execute 31 sec latency 12259.038 ms 1 7627 1.02 MB/sec execute 32 sec latency 13259.272 ms 1 7627 0.99 MB/sec execute 33 sec latency 14259.463 ms 1 7627 0.96 MB/sec execute 34 sec latency 15259.597 ms 1 7892 0.94 MB/sec execute 35 sec latency 15483.599 ms oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8231 0.96 MB/sec execute 36 sec latency 14.787 ms 1 8614 1.02 MB/sec execute 37 sec latency 27.662 ms 1 8940 1.00 MB/sec execute 38 sec latency 15.466 ms 1 9369 0.99 MB/sec execute 39 sec latency 11.247 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4744 1282944 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2684 1285004 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 42564 3560420 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17172 3567836 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 59736 7128256 1% /mnt/lustre 1 9724 1.02 MB/sec execute 40 sec latency 14.276 ms 1 10109 1.10 MB/sec execute 41 sec latency 450.349 ms 1 10735 1.20 MB/sec execute 42 sec latency 14.377 ms 1 11034 1.20 MB/sec execute 43 sec latency 25.349 ms test_26 fail mds1 3 times Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server 1 11357 1.18 MB/sec execute 44 sec latency 18.887 ms 1 11653 1.17 MB/sec execute 45 sec latency 38.020 ms 1 11653 1.14 MB/sec execute 46 sec latency 1038.179 ms 1 11653 1.12 MB/sec execute 47 sec latency 2038.355 ms 1 11653 1.10 MB/sec execute 48 sec latency 3038.511 ms 1 11653 1.07 MB/sec execute 49 sec latency 4038.653 ms 1 11653 1.05 MB/sec execute 50 sec latency 5038.813 ms reboot facets: mds1 1 11653 1.03 MB/sec execute 51 sec latency 6038.998 ms 1 11653 1.01 MB/sec execute 52 sec latency 7039.138 ms 1 11653 0.99 MB/sec execute 53 sec latency 8039.330 ms 1 11653 0.97 MB/sec execute 54 sec latency 9039.478 ms 1 11653 0.96 MB/sec execute 55 sec latency 10039.624 ms 1 11653 0.94 MB/sec execute 56 sec latency 11039.787 ms 1 11653 0.92 MB/sec execute 57 sec latency 12039.983 ms 1 11653 0.91 MB/sec execute 58 sec latency 13040.136 ms 1 11653 0.89 MB/sec execute 59 sec latency 14040.332 ms 1 11653 0.88 MB/sec execute 60 sec latency 15040.504 ms Failover mds1 to oleg201-server 1 11653 0.86 MB/sec execute 61 sec latency 16040.640 ms mount facets: mds1 1 11653 0.85 MB/sec execute 62 sec latency 17040.731 ms 1 11653 0.84 MB/sec execute 63 sec latency 18040.904 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 11653 0.82 MB/sec execute 64 sec latency 19041.081 ms 1 11653 0.81 MB/sec execute 65 sec latency 20041.271 ms 1 11653 0.80 MB/sec execute 66 sec latency 21041.443 ms 1 11653 0.79 MB/sec execute 67 sec latency 22041.678 ms 1 11653 0.77 MB/sec execute 68 sec latency 23041.881 ms oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 1 11653 0.76 MB/sec execute 69 sec latency 24042.044 ms 1 11653 0.75 MB/sec execute 70 sec latency 25042.266 ms 1 11653 0.74 MB/sec execute 71 sec latency 26042.477 ms 1 11653 0.73 MB/sec execute 72 sec latency 27042.661 ms 1 11653 0.72 MB/sec execute 73 sec latency 28042.886 ms 1 11900 0.73 MB/sec execute 74 sec latency 28267.699 ms oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 12274 0.76 MB/sec execute 75 sec latency 17.595 ms 1 12636 0.75 MB/sec execute 76 sec latency 18.988 ms 1 13023 0.76 MB/sec execute 77 sec latency 14.946 ms 1 13532 0.81 MB/sec execute 78 sec latency 16.079 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5424 1282264 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3496 1284192 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 47084 3554868 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 31840 3571072 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 78924 7125940 2% /mnt/lustre 1 14157 0.87 MB/sec execute 79 sec latency 14.002 ms 1 14470 0.88 MB/sec execute 80 sec latency 14.315 ms 1 14718 0.87 MB/sec execute 81 sec latency 22.664 ms 1 15098 0.87 MB/sec execute 82 sec latency 13.131 ms test_26 fail mds2 4 times Failing mds2 on oleg201-server 1 15442 0.88 MB/sec execute 83 sec latency 14.603 ms Stopping /mnt/lustre-mds2 (opts:) on oleg201-server reboot facets: mds2 1 15623 0.90 MB/sec execute 84 sec latency 693.845 ms 1 15623 0.89 MB/sec execute 85 sec latency 1694.016 ms 1 15623 0.88 MB/sec execute 86 sec latency 2694.173 ms 1 15623 0.87 MB/sec execute 87 sec latency 3694.322 ms 1 15623 0.86 MB/sec execute 88 sec latency 4694.508 ms 1 15623 0.85 MB/sec execute 89 sec latency 5694.660 ms 1 15623 0.84 MB/sec execute 90 sec latency 6694.817 ms 1 15623 0.83 MB/sec execute 91 sec latency 7694.983 ms 1 15623 0.82 MB/sec execute 92 sec latency 8695.140 ms 1 15623 0.82 MB/sec execute 93 sec latency 9695.315 ms Failover mds2 to oleg201-server 1 15623 0.81 MB/sec execute 94 sec latency 10695.449 ms mount facets: mds2 1 15623 0.80 MB/sec execute 95 sec latency 11695.619 ms 1 15623 0.79 MB/sec execute 96 sec latency 12695.738 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 15623 0.78 MB/sec execute 97 sec latency 13695.909 ms oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 1 15623 0.77 MB/sec execute 98 sec latency 14696.051 ms Started lustre-MDT0001 1 15623 0.77 MB/sec execute 99 sec latency 15696.261 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 2293 28.380 28267.689 Close 1683 1.418 5.401 Rename 103 11.041 18.137 Unlink 486 4.706 450.323 Qpathinfo 2087 2.000 14.467 Qfileinfo 362 0.373 2.207 Qfsinfo 408 0.670 5.389 Sfileinfo 200 5.461 18.811 Find 823 44.192 19809.881 WriteX 1154 2.038 10.830 ReadX 3747 0.030 0.465 LockX 8 1.262 1.504 UnlockX 8 1.419 1.795 Flush 166 7.216 37.391 Throughput 0.765988 MB/sec 1 clients 1 procs max_latency=28267.699 ms stopping dbench on /mnt/lustre at Fri Jan 17 11:58:18 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Jan 17 11:58:19 EST 2025 waiting for dbench pid 11253 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 207 8.10 MB/sec warmup 1 sec latency 24.354 ms 1 460 8.08 MB/sec warmup 2 sec latency 21.183 ms 1 796 7.16 MB/sec warmup 3 sec latency 32.746 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4328 1283360 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2552 1285136 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36972 3533100 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 4708 3601208 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 41680 7134308 1% /mnt/lustre 1 1131 5.75 MB/sec warmup 4 sec latency 21.230 ms 1 1502 5.21 MB/sec warmup 5 sec latency 30.621 ms 1 1860 4.39 MB/sec warmup 6 sec latency 16.643 ms 1 2247 3.83 MB/sec warmup 7 sec latency 18.130 ms test_26 fail mds1 5 times Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server 1 2596 3.62 MB/sec warmup 8 sec latency 30.018 ms 1 3277 3.93 MB/sec warmup 9 sec latency 40.495 ms 1 3768 3.98 MB/sec warmup 10 sec latency 14.253 ms 1 4001 3.64 MB/sec warmup 11 sec latency 16.174 ms 1 4083 3.34 MB/sec warmup 12 sec latency 692.884 ms 1 4083 3.08 MB/sec warmup 13 sec latency 1693.025 ms 1 4083 2.86 MB/sec warmup 14 sec latency 2693.228 ms reboot facets: mds1 1 4083 2.67 MB/sec warmup 15 sec latency 3693.410 ms 1 4083 2.51 MB/sec warmup 16 sec latency 4693.567 ms 1 4083 2.36 MB/sec warmup 17 sec latency 5693.779 ms 1 4083 2.23 MB/sec warmup 18 sec latency 6693.973 ms 1 4083 2.11 MB/sec warmup 19 sec latency 7694.192 ms 1 4083 0.00 MB/sec execute 1 sec latency 9694.486 ms 1 4083 0.00 MB/sec execute 2 sec latency 10694.632 ms 1 4083 0.00 MB/sec execute 3 sec latency 11694.803 ms 1 4083 0.00 MB/sec execute 4 sec latency 12695.014 ms Failover mds1 to oleg201-server mount facets: mds1 1 4083 0.00 MB/sec execute 5 sec latency 13695.130 ms 1 4083 0.00 MB/sec execute 6 sec latency 14695.261 ms 1 4083 0.00 MB/sec execute 7 sec latency 15695.436 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4083 0.00 MB/sec execute 8 sec latency 16695.600 ms 1 4083 0.00 MB/sec execute 9 sec latency 17695.889 ms 1 4083 0.00 MB/sec execute 10 sec latency 18696.112 ms 1 4083 0.00 MB/sec execute 11 sec latency 19696.238 ms 1 4083 0.00 MB/sec execute 12 sec latency 20696.401 ms oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 1 4083 0.00 MB/sec execute 13 sec latency 21696.558 ms Started lustre-MDT0000 1 4083 0.00 MB/sec execute 14 sec latency 22696.772 ms 1 4083 0.00 MB/sec execute 15 sec latency 23696.946 ms 1 4083 0.00 MB/sec execute 16 sec latency 24697.200 ms 1 4083 0.00 MB/sec execute 17 sec latency 25698.892 ms 1 4083 0.00 MB/sec execute 18 sec latency 26699.913 ms 1 4157 0.00 MB/sec execute 19 sec latency 27325.717 ms 1 4299 0.02 MB/sec execute 20 sec latency 47.237 ms oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4431 0.03 MB/sec execute 21 sec latency 52.666 ms 1 4528 0.04 MB/sec execute 22 sec latency 57.253 ms 1 4691 0.09 MB/sec execute 23 sec latency 59.122 ms 1 4957 0.21 MB/sec execute 24 sec latency 30.057 ms 1 5104 0.21 MB/sec execute 25 sec latency 72.170 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3440 1284248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2560 1285128 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 40532 3566432 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13956 3592644 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 54488 7159076 1% /mnt/lustre 1 5190 0.20 MB/sec execute 26 sec latency 71.919 ms 1 5317 0.20 MB/sec execute 27 sec latency 44.621 ms 1 5430 0.19 MB/sec execute 28 sec latency 33.815 ms 1 5571 0.19 MB/sec execute 29 sec latency 73.976 ms 1 5763 0.19 MB/sec execute 30 sec latency 25.844 ms test_26 fail mds2 6 times 1 6002 0.23 MB/sec execute 31 sec latency 31.779 ms Failing mds2 on oleg201-server Stopping /mnt/lustre-mds2 (opts:) on oleg201-server 1 6166 0.25 MB/sec execute 32 sec latency 450.754 ms reboot facets: mds2 1 6166 0.24 MB/sec execute 33 sec latency 1450.865 ms 1 6166 0.24 MB/sec execute 34 sec latency 2450.985 ms 1 6166 0.23 MB/sec execute 35 sec latency 3451.099 ms 1 6166 0.22 MB/sec execute 36 sec latency 4451.286 ms 1 6166 0.22 MB/sec execute 37 sec latency 5451.458 ms 1 6166 0.21 MB/sec execute 38 sec latency 6451.649 ms 1 6166 0.21 MB/sec execute 39 sec latency 7451.844 ms 1 6166 0.20 MB/sec execute 40 sec latency 8452.694 ms 1 6166 0.20 MB/sec execute 41 sec latency 9454.395 ms 1 6166 0.19 MB/sec execute 42 sec latency 10455.390 ms Failover mds2 to oleg201-server mount facets: mds2 1 6166 0.19 MB/sec execute 43 sec latency 11455.633 ms 1 6166 0.18 MB/sec execute 44 sec latency 12455.791 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 6166 0.18 MB/sec execute 45 sec latency 13455.970 ms 1 6166 0.17 MB/sec execute 46 sec latency 14456.166 ms oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 1 6166 0.17 MB/sec execute 47 sec latency 15456.326 ms 1 6166 0.17 MB/sec execute 48 sec latency 16456.483 ms 1 6166 0.16 MB/sec execute 49 sec latency 17456.664 ms 1 6166 0.16 MB/sec execute 50 sec latency 18456.824 ms 1 6477 0.25 MB/sec execute 51 sec latency 18982.974 ms oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 6916 0.28 MB/sec execute 52 sec latency 19.310 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7241 0.34 MB/sec execute 53 sec latency 27.295 ms striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar 1 7423 0.35 MB/sec execute 54 sec latency 32.804 ms 1 7603 0.35 MB/sec execute 55 sec latency 20.517 ms 1 7818 0.35 MB/sec execute 56 sec latency 26.409 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1285300 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1285568 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 38156 3567872 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11772 3594720 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49928 7162592 1% /mnt/lustre 1 7999 0.35 MB/sec execute 57 sec latency 41.063 ms 1 8087 0.35 MB/sec execute 58 sec latency 416.695 ms 1 8268 0.36 MB/sec execute 59 sec latency 502.787 ms 1 8582 0.40 MB/sec execute 60 sec latency 21.222 ms test_26 fail mds1 7 times Failing mds1 on oleg201-server 1 8799 0.40 MB/sec execute 61 sec latency 30.716 ms Stopping /mnt/lustre-mds1 (opts:) on oleg201-server 1 8922 0.39 MB/sec execute 62 sec latency 454.473 ms reboot facets: mds1 1 8922 0.39 MB/sec execute 63 sec latency 1454.692 ms 1 8922 0.38 MB/sec execute 64 sec latency 2454.859 ms 1 8922 0.38 MB/sec execute 65 sec latency 3455.074 ms 1 8922 0.37 MB/sec execute 66 sec latency 4455.234 ms 1 8922 0.37 MB/sec execute 67 sec latency 5455.399 ms 1 8922 0.36 MB/sec execute 68 sec latency 6455.621 ms 1 8922 0.35 MB/sec execute 69 sec latency 7456.243 ms 1 8922 0.35 MB/sec execute 70 sec latency 8456.442 ms 1 8922 0.34 MB/sec execute 71 sec latency 9456.699 ms 1 8922 0.34 MB/sec execute 72 sec latency 10456.992 ms Failover mds1 to oleg201-server mount facets: mds1 1 8922 0.34 MB/sec execute 73 sec latency 11457.192 ms 1 8922 0.33 MB/sec execute 74 sec latency 12457.378 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 8922 0.33 MB/sec execute 75 sec latency 13457.634 ms 1 8922 0.32 MB/sec execute 76 sec latency 14457.787 ms 1 8922 0.32 MB/sec execute 77 sec latency 15457.955 ms 1 8922 0.31 MB/sec execute 78 sec latency 16458.120 ms 1 8922 0.31 MB/sec execute 79 sec latency 17458.279 ms oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 1 8922 0.31 MB/sec execute 80 sec latency 18458.478 ms 1 8922 0.30 MB/sec execute 81 sec latency 19458.622 ms 1 8922 0.30 MB/sec execute 82 sec latency 20458.760 ms 1 8922 0.29 MB/sec execute 83 sec latency 21458.903 ms 1 8929 0.29 MB/sec execute 84 sec latency 22420.557 ms oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 9307 0.29 MB/sec execute 85 sec latency 13.943 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 9684 0.32 MB/sec execute 86 sec latency 15.452 ms 1 10355 0.38 MB/sec execute 87 sec latency 15.210 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 10875 0.43 MB/sec execute 88 sec latency 12.624 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Jan 17 12:00:08 EST 2025 with return code 0 11253 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 11253 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (240s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 12:00:13 (1737133213) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00295987 s, 1.4 MB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg201-server Stopping /mnt/lustre-ost1 (opts:) on oleg201-server reboot facets: ost1 Failover ost1 to oleg201-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00563992 s, 726 kB/s pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (36s) == replay-dual test 29: replay vs update with the same xid ========================================================== 12:00:49 (1737133249) SKIP: replay-dual test_29 needs >= 2 clients SKIP 29 (2s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 12:00:51 (1737133251) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0109994 s, 3.7 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0113452 s, 3.6 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server reboot facets: mds1 Failover mds1 to oleg201-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 160+0 records in 160+0 records out 81920 bytes (82 kB) copied, 24.4193 s, 3.4 kB/s oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (30s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 12:01:21 (1737133281) Failing ost1 on oleg201-server Stopping /mnt/lustre-ost1 (opts:) on oleg201-server striped dir -i1 -c2 -H all_char /mnt/lustre/d31.replay-dual/lustre-OST0000 reboot facets: ost1 Creating to objid 3297 on ost lustre-OST0000... total: 32 open/close in 0.19 seconds: 164.69 ops/second at_max=0 fail_loc=0x80001420 Failover ost1 to oleg201-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL IDLE state after 0 sec at_max=600 PASS 31 (29s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 12:01:51 (1737133311) fail_loc=0x0000131d fail_val=10 fail_loc=0x726 Stopping /mnt/lustre-mds2 (opts:) on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server fail_loc=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2468 1285192 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2088 1285600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11784 3594036 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 13196 7199644 1% /mnt/lustre PASS 32 (18s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 12:02:08 (1737133328) Stopping /mnt/lustre-mds1 (opts:) on oleg201-server last_rcvd in /dev/mapper/mds1_flakey: incompat = 0x61c Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount (REPLAY_WAIT|REPLAY_LOCKS) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in REPLAY_LOCKS state after 0 sec oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping /mnt/lustre-mds1 (opts:) on oleg201-server last_rcvd in /dev/mapper/mds1_flakey: incompat = 0x61c Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre2 mount.lustre: mount oleg201-server@tcp:/lustre at /mnt/lustre2 failed: Input/output error Is the MGS running? oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec PASS 33 (32s) == replay-dual test complete, duration 2767 sec ========== 12:02:40 (1737133360) Stopping clients: oleg201-client.virtnet /mnt/lustre2 (opts:)