-----============= acceptance-small: replay-dual ============----- Fri Oct 24 09:12:16 EDT 2025 oleg337-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b Starting client oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 Started clients oleg337-client.virtnet: 192.168.203.137@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) oleg337-client.virtnet: executing check_config_client /mnt/lustre oleg337-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg337-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b41c2ef1800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9b41c5b28800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b41c2ef1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b41c5b28800.idle_timeout=debug disable quota as required oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 osd-ldiskfs.track_declares_assert=1 == replay-dual test 0a: expired recovery with lost client ========================================================== 09:12:55 (1761311575) Check file is LU482_FAILED=/tmp/replay-dual.lu482.eXDAdo UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 50 open/close in 1.07 seconds: 46.89 ops/second fail_loc=0x80000514 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1761311694 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (126s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 09:15:01 (1761311701) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 PASS 0b (139s) == replay-dual test 1: |X| simple create ================= 09:17:20 (1761311840) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg337-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (56s) == replay-dual test 2: |X| mkdir adir ==================== 09:18:16 (1761311896) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (56s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 09:19:12 (1761311952) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (56s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 09:20:08 (1761312008) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1920 1285768 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1760 1285928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (54s) == replay-dual test 5: open, unlink |X| close ============ 09:21:02 (1761312062) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7731 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (50s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 09:21:52 (1761312112) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7731 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7731 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (53s) == replay-dual test 8: replay of resent request ========== 09:22:45 (1761312165) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (103s) == replay-dual test 9: resending a replayed create ======= 09:24:28 (1761312268) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (97s) == replay-dual test 10: resending a replayed unlink ====== 09:26:05 (1761312365) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (95s) == replay-dual test 11: both clients timeout during replay ========================================================== 09:27:40 (1761312460) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 34 sec fail_loc=0 PASS 11 (87s) == replay-dual test 12: open resend timeout ============== 09:29:07 (1761312547) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7731 fail_loc=0x80000302 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (90s) == replay-dual test 13: close resend timeout ============= 09:30:37 (1761312637) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7731 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (46s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 09:31:25 (1761312685) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.49 seconds: 51.37 ops/second total: 1 open/close in 0.02 seconds: 46.02 ops/second Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761312829 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 PASS 15a (149s) == replay-dual test 15c: remove multiple OST orphans ===== 09:33:54 (1761312834) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 PASS 15c (197s) == replay-dual test 16: fail MDS during recovery (3571) == 09:37:11 (1761313031) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.45 seconds: 55.69 ops/second total: 1 open/close in 0.01 seconds: 78.88 ops/second Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761313219 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 PASS 16 (193s) == replay-dual test 17: fail OST during recovery (3571) == 09:40:24 (1761313224) total: 25 open/close in 0.43 seconds: 57.92 ops/second total: 1 open/close in 0.03 seconds: 34.19 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing ost1 on oleg337-server Stopping /mnt/lustre-ost1 (opts:) on oleg337-server reboot facets: ost1 Failover ost1 to oleg337-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg337-server Stopping /mnt/lustre-ost1 (opts:) on oleg337-server reboot facets: ost1 Failover ost1 to oleg337-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761313368 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 PASS 17 (149s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 09:42:53 (1761313373) debug=+dlmtrace using seed 433849113 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.203.137@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9b41c70a3000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9b41c8f5e000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff9b41c70a3000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff9b41c8f5e000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9b41c70a3000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9b41c8f5e000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9b41c70a3000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9b41c8f5e000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.203.137@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9b41c70a3000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9b41c8f5e000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff9b41c70a3000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff9b41c8f5e000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9b41c70a3000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9b41c8f5e000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9b41c70a3000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9b41c8f5e000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (53s) == replay-dual test 19: resend of open request =========== 09:43:46 (1761313426) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1761313528.97 total 94.28 last 0.00) total: 1 open/close in 94.28 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (153s) == replay-dual test 20: recovery time is not increasing == 09:46:19 (1761313579) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 PASS 20 (391s) == replay-dual test 21a: commit on sharing =============== 09:52:50 (1761313970) mdt.lustre-MDT0000.commit_on_sharing=1 mdt.lustre-MDT0001.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 mdt.lustre-MDT0001.commit_on_sharing=0 PASS 21a (193s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 09:56:04 (1761314164) fail_loc=0x119 Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server reboot facets: mds2 Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1896 1285792 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1752 1285936 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.09 seconds: 21.17 ops/second total: 2 open/close in 0.03 seconds: 62.01 ops/second Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22a (92s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 09:57:36 (1761314256) fail_loc=0x119 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.06 seconds: 31.80 ops/second total: 2 open/close in 0.04 seconds: 48.89 ops/second Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22b (119s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 09:59:35 (1761314375) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.06 seconds: 32.94 ops/second total: 2 open/close in 0.03 seconds: 66.26 ops/second Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22c (94s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 10:01:09 (1761314469) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1712 1285976 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.09 seconds: 22.62 ops/second total: 2 open/close in 0.05 seconds: 36.92 ops/second Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22d (132s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 10:03:21 (1761314601) fail_loc=0x119 fail_loc=0 Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server reboot facets: mds2 Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.05 seconds: 37.22 ops/second Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23a (90s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 10:04:51 (1761314691) fail_loc=0x119 fail_loc=0 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 45.30 ops/second Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23b (111s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 10:06:42 (1761314802) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 53.14 ops/second Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23c (101s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 10:08:23 (1761314903) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1708 1285980 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.05 seconds: 41.76 ops/second Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23d (119s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 10:10:22 (1761315022) touch: cannot touch '/mnt/lustre/f24.replay-dual': No space left on device fail_loc=0x119 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory fail_loc=0 rm: cannot remove '/mnt/lustre2/f24.replay-dual': No such file or directory PASS 24 (50s) == replay-dual test 25: replay|resend ==================== 10:11:13 (1761315073) lfs setstripe: setstripe error for '/mnt/lustre/f25.replay-dual': No space left on device dd: failed to open '/mnt/lustre/f25.replay-dual': No space left on device replay-dual test_25: @@@@@@ FAIL: failed to write data Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6482:error() = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:912:test_25() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6840:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6887:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6728:run_test() = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:935:main() Dumping lctl log to /tmp/testlogs//replay-dual.test_25.*.1761315219.log rsync: chown "/tmp/testlogs/.replay-dual.test_25.debug_log.oleg337-server.1761315219.log.DTrbhK" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-dual.test_25.dmesg.oleg337-server.1761315219.log.j0iKa7" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 25 (150s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 10:13:42 (1761315222) Starting client oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre Started clients oleg337-client.virtnet: 192.168.203.137@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started tar loop with pid 65904 Started dbench loop with 65905 striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt tar: etc/resolv.conf: Cannot open: No space left on device '/usr/share/dbench/client.txt' -> 'client.txt' cp: cannot create regular file 'client.txt': No space left on device UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1964 1285724 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1808 1285880 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] stopping dbench on /mnt/lustre at Fri Oct 24 10:14:25 EDT 2025 with return code 1 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre /mnt/lustre dbench exit with error 1 replay-dual test_26: @@@@@@ FAIL: dbench failed with rc 1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6482:error() = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:1033:test_26() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6840:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6887:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6728:run_test() = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:1072:main() Dumping lctl log to /tmp/testlogs//replay-dual.test_26.*.1761315267.log rsync: chown "/tmp/testlogs/.replay-dual.test_26.debug_log.oleg337-server.1761315267.log.PxUgjz" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-dual.test_26.dmesg.oleg337-server.1761315267.log.WJqKbT" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] test_26 fail mds1 1 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/replay-dual.sh: line 1051: kill: (65905) - No such process replay-dual test_26: @@@@@@ FAIL: dbench 65905 missing Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6482:error() = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:1051:test_26() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6840:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6887:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6728:run_test() = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:1072:main() Dumping lctl log to /tmp/testlogs//replay-dual.test_26.*.1761315310.log rsync: chown "/tmp/testlogs/.replay-dual.test_26.debug_log.oleg337-server.1761315310.log.0DF2dN" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-dual.test_26.dmesg.oleg337-server.1761315310.log.VLy3ad" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now /home/green/git/lustre-release/lustre/tests/replay-dual.sh: line 945: kill: (65905) - No such process FAIL 26 (97s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 10:15:20 (1761315320) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00760167 s, 539 kB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg337-server Stopping /mnt/lustre-ost1 (opts:) on oleg337-server reboot facets: ost1 Failover ost1 to oleg337-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00742406 s, 552 kB/s oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (47s) == replay-dual test 29: replay vs update with the same xid ========================================================== 10:16:07 (1761315367) SKIP: replay-dual test_29 needs >= 2 clients SKIP 29 (2s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 10:16:09 (1761315369) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0153813 s, 2.7 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0096887 s, 4.2 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server reboot facets: mds1 Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 160+0 records in 160+0 records out 81920 bytes (82 kB, 80 KiB) copied, 33.8129 s, 2.4 kB/s oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (48s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 10:16:57 (1761315417) Failing ost1 on oleg337-server Stopping /mnt/lustre-ost1 (opts:) on oleg337-server striped dir -i1 -c2 -H crush /mnt/lustre/d31.replay-dual/lustre-OST0000 Creating to objid 2241 on ost lustre-OST0000... reboot facets: ost1 total: 32 open/close in 0.71 seconds: 44.80 ops/second at_max=0 fail_loc=0x80001420 Failover ost1 to oleg337-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL IDLE state after 0 sec at_max=600 PASS 31 (40s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 10:17:38 (1761315458) fail_loc=0x0000131d fail_val=10 fail_loc=0x726 Stopping /mnt/lustre-mds2 (opts:) on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server fail_loc=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2200 1285488 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2132 3603904 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3540 7209516 1% /mnt/lustre PASS 32 (40s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 10:18:18 (1761315498) at_min=60 Stopping /mnt/lustre-mds2 (opts:) on oleg337-server last_rcvd in /dev/mapper/mds2_flakey: incompat = 0x61c Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount REPLAY_WAIT mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in REPLAY_WAIT state after 0 sec oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 oleg337-server: *.lustre-MDT0001.recovery_status status: COMPLETE Stopping /mnt/lustre-mds2 (opts:) on oleg337-server last_rcvd in /dev/mapper/mds2_flakey: incompat = 0x61c Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 Starting client: oleg337-client.virtnet: -o user_xattr,flock oleg337-server@tcp:/lustre /mnt/lustre2 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds2 oleg337-server: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 oleg337-server: *.lustre-MDT0001.recovery_status status: COMPLETE at_min=0 PASS 33 (71s) == replay-dual test complete, duration 4033 sec ========== 10:19:30 (1761315570) replay-dual: FAIL: test_25 failed to write data replay-dual: FAIL: test_26 dbench failed with rc 1 replay-dual: FAIL: test_26 dbench 65905 missing Stopping clients: oleg337-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg337-client.virtnet /mnt/lustre2 opts: