-----============= acceptance-small: replay-dual ============----- Fri Oct 24 11:05:03 EDT 2025 oleg211-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b Starting client oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 Started clients oleg211-client.virtnet: 192.168.202.111@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) oleg211-client.virtnet: executing check_config_client /mnt/lustre oleg211-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg211-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ae8c38b9800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9ae8c9a5d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ae8c38b9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ae8c9a5d000.idle_timeout=debug disable quota as required oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 osd-ldiskfs.track_declares_assert=1 == replay-dual test 0a: expired recovery with lost client ========================================================== 11:05:33 (1761318333) Check file is LU482_FAILED=/tmp/replay-dual.lu482.l0qv4H UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 50 open/close in 0.82 seconds: 60.95 ops/second fail_loc=0x80000514 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1761318448 ; total 1 ; last 1) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (119s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 11:07:32 (1761318452) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 PASS 0b (141s) == replay-dual test 1: |X| simple create ================= 11:09:53 (1761318593) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg211-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (54s) == replay-dual test 2: |X| mkdir adir ==================== 11:10:47 (1761318647) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (54s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 11:11:41 (1761318701) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (49s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 11:12:30 (1761318750) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1760 1285928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (44s) == replay-dual test 5: open, unlink |X| close ============ 11:13:14 (1761318794) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7712 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (38s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 11:13:52 (1761318832) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7712 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7712 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (47s) == replay-dual test 8: replay of resent request ========== 11:14:39 (1761318879) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (53s) == replay-dual test 9: resending a replayed create ======= 11:15:32 (1761318932) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (46s) == replay-dual test 10: resending a replayed unlink ====== 11:16:18 (1761318978) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (60s) == replay-dual test 11: both clients timeout during replay ========================================================== 11:17:18 (1761319038) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 13 sec fail_loc=0 PASS 11 (56s) == replay-dual test 12: open resend timeout ============== 11:18:14 (1761319094) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7712 fail_loc=0x80000302 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (39s) == replay-dual test 13: close resend timeout ============= 11:18:53 (1761319133) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7712 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (40s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 11:19:33 (1761319173) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.34 seconds: 73.84 ops/second total: 1 open/close in 0.01 seconds: 70.37 ops/second Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761319275 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 PASS 15a (105s) == replay-dual test 15c: remove multiple OST orphans ===== 11:21:18 (1761319278) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 PASS 15c (155s) == replay-dual test 16: fail MDS during recovery (3571) == 11:23:53 (1761319433) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.31 seconds: 80.55 ops/second total: 1 open/close in 0.01 seconds: 91.60 ops/second Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761319575 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 PASS 16 (145s) == replay-dual test 17: fail OST during recovery (3571) == 11:26:18 (1761319578) total: 25 open/close in 0.29 seconds: 85.96 ops/second total: 1 open/close in 0.03 seconds: 39.42 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing ost1 on oleg211-server Stopping /mnt/lustre-ost1 (opts:) on oleg211-server reboot facets: ost1 Failover ost1 to oleg211-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg211-server Stopping /mnt/lustre-ost1 (opts:) on oleg211-server reboot facets: ost1 Failover ost1 to oleg211-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761319767 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 PASS 17 (194s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 11:29:32 (1761319772) debug=+dlmtrace using seed 1063303812 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.202.111@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9ae8c9a5c800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9ae8e010c000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff9ae8c9a5c800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff9ae8e010c000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9ae8c9a5c800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9ae8e010c000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9ae8c9a5c800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9ae8e010c000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.202.111@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9ae8c9a5c800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9ae8e010c000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff9ae8c9a5c800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff9ae8e010c000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9ae8c9a5c800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9ae8e010c000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9ae8c9a5c800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9ae8e010c000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (52s) == replay-dual test 19: resend of open request =========== 11:30:24 (1761319824) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1761319912.25 total 78.55 last 0.00) total: 1 open/close in 78.55 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (134s) == replay-dual test 20: recovery time is not increasing == 11:32:38 (1761319958) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 PASS 20 (359s) == replay-dual test 21a: commit on sharing =============== 11:38:37 (1761320317) mdt.lustre-MDT0000.commit_on_sharing=1 mdt.lustre-MDT0001.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 mdt.lustre-MDT0001.commit_on_sharing=0 PASS 21a (169s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 11:41:27 (1761320487) fail_loc=0x119 Failing mds2 on oleg211-server Stopping /mnt/lustre-mds2 (opts:) on oleg211-server reboot facets: mds2 Failover mds2 to oleg211-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1896 1285792 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.06 seconds: 35.86 ops/second total: 2 open/close in 0.03 seconds: 78.93 ops/second Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22a (79s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 11:42:46 (1761320566) fail_loc=0x119 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server Failing mds2 on oleg211-server Stopping /mnt/lustre-mds2 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg211-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.06 seconds: 32.63 ops/second total: 2 open/close in 0.03 seconds: 62.05 ops/second Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22b (101s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 11:44:27 (1761320667) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.05 seconds: 38.29 ops/second total: 2 open/close in 0.03 seconds: 58.37 ops/second Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22c (76s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 11:45:43 (1761320743) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server Failing mds2 on oleg211-server Stopping /mnt/lustre-mds2 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg211-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1712 1285976 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.05 seconds: 42.73 ops/second total: 2 open/close in 0.02 seconds: 81.84 ops/second Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22d (103s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 11:47:26 (1761320846) fail_loc=0x119 fail_loc=0 Failing mds2 on oleg211-server Stopping /mnt/lustre-mds2 (opts:) on oleg211-server reboot facets: mds2 Failover mds2 to oleg211-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 54.86 ops/second Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23a (70s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 11:48:37 (1761320917) fail_loc=0x119 fail_loc=0 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server Failing mds2 on oleg211-server Stopping /mnt/lustre-mds2 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg211-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 51.43 ops/second Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23b (84s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 11:50:01 (1761321001) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 53.48 ops/second Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23c (74s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 11:51:15 (1761321075) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server Failing mds2 on oleg211-server Stopping /mnt/lustre-mds2 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg211-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1708 1285980 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 66.99 ops/second Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23d (111s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 11:53:06 (1761321186) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (81s) == replay-dual test 25: replay|resend ==================== 11:54:27 (1761321267) 1+0 records in 1+0 records out 512 bytes copied, 0.00542724 s, 94.3 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg211-server Stopping /mnt/lustre-ost1 (opts:) on oleg211-server reboot facets: ost1 Failover ost1 to oleg211-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6009: 65047 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (27s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 11:54:54 (1761321294) Starting client oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre Started clients oleg211-client.virtnet: 192.168.202.111@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started tar loop with pid 66529 Started dbench loop with 66530 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Oct 24 11:54:57 EDT 2025 waiting for dbench pid 66573 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3580628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2796 7186240 1% /mnt/lustre 1 113 5.24 MB/sec warmup 1 sec latency 47.318 ms 1 211 4.08 MB/sec warmup 2 sec latency 56.911 ms 1 313 3.83 MB/sec warmup 3 sec latency 48.117 ms 1 421 3.72 MB/sec warmup 4 sec latency 75.047 ms 1 523 3.61 MB/sec warmup 5 sec latency 40.287 ms test_26 fail mds1 1 times Failing mds1 on oleg211-server 1 705 3.53 MB/sec warmup 6 sec latency 61.996 ms Stopping /mnt/lustre-mds1 (opts:) on oleg211-server 1 786 3.06 MB/sec warmup 7 sec latency 164.684 ms 1 910 2.72 MB/sec warmup 8 sec latency 91.753 ms 1 995 2.43 MB/sec warmup 9 sec latency 398.954 ms 1 995 2.19 MB/sec warmup 10 sec latency 1399.923 ms 1 995 1.99 MB/sec warmup 11 sec latency 2400.223 ms 1 995 1.83 MB/sec warmup 12 sec latency 3400.462 ms 1 995 1.68 MB/sec warmup 13 sec latency 4401.158 ms reboot facets: mds1 1 995 1.56 MB/sec warmup 14 sec latency 5401.388 ms 1 995 1.46 MB/sec warmup 15 sec latency 6401.572 ms 1 995 1.37 MB/sec warmup 16 sec latency 7401.804 ms 1 995 1.29 MB/sec warmup 17 sec latency 8402.022 ms 1 995 1.22 MB/sec warmup 18 sec latency 9402.228 ms 1 995 1.15 MB/sec warmup 19 sec latency 10402.468 ms 1 995 0.00 MB/sec execute 1 sec latency 12402.897 ms 1 995 0.00 MB/sec execute 2 sec latency 13403.143 ms 1 995 0.00 MB/sec execute 3 sec latency 14403.728 ms Failover mds1 to oleg211-server 1 995 0.00 MB/sec execute 4 sec latency 15403.996 ms mount facets: mds1 1 995 0.00 MB/sec execute 5 sec latency 16404.393 ms 1 995 0.00 MB/sec execute 6 sec latency 17404.606 ms 1 995 0.00 MB/sec execute 7 sec latency 18404.905 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.00 MB/sec execute 8 sec latency 19405.155 ms 1 995 0.00 MB/sec execute 9 sec latency 20405.528 ms 1 995 0.00 MB/sec execute 10 sec latency 21405.800 ms oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 1 995 0.00 MB/sec execute 11 sec latency 22406.063 ms pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 1 995 0.00 MB/sec execute 12 sec latency 23406.221 ms 1 995 0.00 MB/sec execute 13 sec latency 24408.087 ms 1 995 0.00 MB/sec execute 14 sec latency 25408.246 ms 1 995 0.00 MB/sec execute 15 sec latency 26409.719 ms 1 1000 0.01 MB/sec execute 16 sec latency 27209.649 ms 1 1134 0.06 MB/sec execute 17 sec latency 155.399 ms oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1273 0.07 MB/sec execute 18 sec latency 32.573 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1499 0.22 MB/sec execute 19 sec latency 33.314 ms 1 1671 0.22 MB/sec execute 20 sec latency 39.610 ms 1 1843 0.21 MB/sec execute 21 sec latency 36.590 ms 1 2042 0.21 MB/sec execute 22 sec latency 28.274 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2224 1285464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13760 3592300 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 45980 3555328 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 59740 7147628 1% /mnt/lustre 1 2190 0.21 MB/sec execute 23 sec latency 40.581 ms 1 2355 0.25 MB/sec execute 24 sec latency 33.780 ms 1 2467 0.27 MB/sec execute 25 sec latency 31.763 ms 1 2617 0.28 MB/sec execute 26 sec latency 40.933 ms 1 2935 0.43 MB/sec execute 27 sec latency 35.739 ms test_26 fail mds2 2 times 1 3232 0.44 MB/sec execute 28 sec latency 37.628 ms Failing mds2 on oleg211-server 1 3395 0.47 MB/sec execute 29 sec latency 30.138 ms Stopping /mnt/lustre-mds2 (opts:) on oleg211-server 1 3633 0.56 MB/sec execute 30 sec latency 26.115 ms 1 3785 0.58 MB/sec execute 31 sec latency 36.507 ms 1 3897 0.57 MB/sec execute 32 sec latency 37.611 ms reboot facets: mds2 1 3988 0.55 MB/sec execute 33 sec latency 53.067 ms 1 4083 0.53 MB/sec execute 34 sec latency 142.716 ms 1 4083 0.52 MB/sec execute 35 sec latency 1143.435 ms 1 4083 0.50 MB/sec execute 36 sec latency 2143.836 ms 1 4083 0.49 MB/sec execute 37 sec latency 3144.193 ms 1 4083 0.48 MB/sec execute 38 sec latency 4144.512 ms 1 4083 0.47 MB/sec execute 39 sec latency 5145.346 ms 1 4083 0.45 MB/sec execute 40 sec latency 6145.687 ms 1 4083 0.44 MB/sec execute 41 sec latency 7146.015 ms 1 4083 0.43 MB/sec execute 42 sec latency 8146.346 ms Failover mds2 to oleg211-server 1 4083 0.42 MB/sec execute 43 sec latency 9146.558 ms mount facets: mds2 1 4083 0.41 MB/sec execute 44 sec latency 10146.830 ms 1 4083 0.40 MB/sec execute 45 sec latency 11147.243 ms 1 4083 0.40 MB/sec execute 46 sec latency 12147.447 ms 1 4083 0.39 MB/sec execute 47 sec latency 13148.463 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4083 0.38 MB/sec execute 48 sec latency 14148.746 ms 1 4083 0.37 MB/sec execute 49 sec latency 15149.046 ms oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 1 4083 0.36 MB/sec execute 50 sec latency 16149.487 ms Started lustre-MDT0001 1 4083 0.36 MB/sec execute 51 sec latency 17149.905 ms 1 4083 0.35 MB/sec execute 52 sec latency 18150.219 ms 1 4083 0.34 MB/sec execute 53 sec latency 19150.560 ms 1 4101 0.34 MB/sec execute 54 sec latency 19986.735 ms 1 4248 0.33 MB/sec execute 55 sec latency 46.072 ms oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4369 0.33 MB/sec execute 56 sec latency 50.985 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4491 0.33 MB/sec execute 57 sec latency 50.107 ms 1 4645 0.35 MB/sec execute 58 sec latency 28.865 ms 1 4787 0.34 MB/sec execute 59 sec latency 39.880 ms 1 4976 0.39 MB/sec execute 60 sec latency 65.757 ms 1 5114 0.38 MB/sec execute 61 sec latency 36.216 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2996 1284692 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2844 1284844 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14556 3591236 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 55068 3549316 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 69624 7140552 1% /mnt/lustre 1 5218 0.38 MB/sec execute 62 sec latency 36.734 ms 1 5358 0.37 MB/sec execute 63 sec latency 36.924 ms 1 5539 0.37 MB/sec execute 64 sec latency 31.116 ms 1 5701 0.37 MB/sec execute 65 sec latency 30.471 ms test_26 fail mds1 3 times 1 5900 0.38 MB/sec execute 66 sec latency 25.326 ms Failing mds1 on oleg211-server 1 6013 0.38 MB/sec execute 67 sec latency 25.869 ms Stopping /mnt/lustre-mds1 (opts:) on oleg211-server 1 6115 0.38 MB/sec execute 68 sec latency 530.133 ms 1 6115 0.38 MB/sec execute 69 sec latency 1530.348 ms 1 6115 0.37 MB/sec execute 70 sec latency 2530.742 ms 1 6115 0.37 MB/sec execute 71 sec latency 3531.080 ms 1 6115 0.36 MB/sec execute 72 sec latency 4531.329 ms 1 6115 0.36 MB/sec execute 73 sec latency 5531.622 ms reboot facets: mds1 1 6115 0.35 MB/sec execute 74 sec latency 6532.144 ms 1 6115 0.35 MB/sec execute 75 sec latency 7532.367 ms 1 6115 0.34 MB/sec execute 76 sec latency 8532.620 ms 1 6115 0.34 MB/sec execute 77 sec latency 9532.868 ms 1 6115 0.33 MB/sec execute 78 sec latency 10533.123 ms 1 6115 0.33 MB/sec execute 79 sec latency 11533.379 ms 1 6115 0.33 MB/sec execute 80 sec latency 12533.585 ms 1 6115 0.32 MB/sec execute 81 sec latency 13533.799 ms 1 6115 0.32 MB/sec execute 82 sec latency 14534.069 ms 1 6115 0.31 MB/sec execute 83 sec latency 15534.278 ms 1 6115 0.31 MB/sec execute 84 sec latency 16534.473 ms Failover mds1 to oleg211-server mount facets: mds1 1 6115 0.31 MB/sec execute 85 sec latency 17534.731 ms 1 6115 0.30 MB/sec execute 86 sec latency 18534.969 ms 1 6115 0.30 MB/sec execute 87 sec latency 19535.322 ms 1 6115 0.30 MB/sec execute 88 sec latency 20535.471 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 6115 0.29 MB/sec execute 89 sec latency 21535.733 ms 1 6115 0.29 MB/sec execute 90 sec latency 22535.954 ms 1 6115 0.29 MB/sec execute 91 sec latency 23536.178 ms 1 6115 0.28 MB/sec execute 92 sec latency 24536.508 ms 1 6115 0.28 MB/sec execute 93 sec latency 25536.746 ms 1 6115 0.28 MB/sec execute 94 sec latency 26537.116 ms 1 6115 0.27 MB/sec execute 95 sec latency 27537.355 ms 1 6115 0.27 MB/sec execute 96 sec latency 28537.553 ms 1 6115 0.27 MB/sec execute 97 sec latency 29537.799 ms 1 6115 0.27 MB/sec execute 98 sec latency 30538.079 ms oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 1 6115 0.26 MB/sec execute 99 sec latency 31538.365 ms pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 889 48.726 27209.635 Close 685 55.759 35543.762 Rename 37 24.544 50.091 Unlink 151 9.581 34.752 Qpathinfo 878 27.437 19986.702 Qfileinfo 151 0.866 4.508 Qfsinfo 128 0.627 6.183 Sfileinfo 80 13.505 23.218 Find 306 4.390 26.438 WriteX 452 4.957 24.926 ReadX 1307 0.416 62.456 LockX 2 3.628 4.455 UnlockX 2 3.547 3.899 Flush 53 18.182 40.917 Throughput 0.263736 MB/sec 1 clients 1 procs max_latency=31538.365 ms stopping dbench on /mnt/lustre at Fri Oct 24 11:57:02 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Oct 24 11:57:06 EDT 2025 waiting for dbench pid 69965 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 129 5.76 MB/sec warmup 1 sec latency 48.938 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3548 1284140 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3444 1284244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 3000 3602808 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 48088 3558108 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51088 7160916 1% /mnt/lustre 1 229 4.41 MB/sec warmup 2 sec latency 49.085 ms 1 337 4.07 MB/sec warmup 3 sec latency 42.161 ms 1 451 3.96 MB/sec warmup 4 sec latency 48.449 ms 1 608 4.19 MB/sec warmup 5 sec latency 68.779 ms 1 769 3.56 MB/sec warmup 6 sec latency 27.757 ms test_26 fail mds2 4 times 1 900 3.10 MB/sec warmup 7 sec latency 42.513 ms Failing mds2 on oleg211-server 1 1033 2.86 MB/sec warmup 8 sec latency 39.231 ms Stopping /mnt/lustre-mds2 (opts:) on oleg211-server 1 1177 2.56 MB/sec warmup 9 sec latency 47.656 ms reboot facets: mds2 1 1278 2.31 MB/sec warmup 10 sec latency 273.044 ms 1 1278 2.10 MB/sec warmup 11 sec latency 1273.262 ms 1 1278 1.93 MB/sec warmup 12 sec latency 2273.536 ms 1 1278 1.78 MB/sec warmup 13 sec latency 3273.772 ms 1 1278 1.65 MB/sec warmup 14 sec latency 4274.075 ms 1 1278 1.54 MB/sec warmup 15 sec latency 5274.332 ms 1 1278 1.45 MB/sec warmup 16 sec latency 6274.628 ms 1 1278 1.36 MB/sec warmup 17 sec latency 7274.867 ms 1 1278 1.29 MB/sec warmup 18 sec latency 8275.093 ms 1 1278 1.22 MB/sec warmup 19 sec latency 9275.295 ms Failover mds2 to oleg211-server mount facets: mds2 1 1278 0.00 MB/sec execute 1 sec latency 11275.672 ms 1 1278 0.00 MB/sec execute 2 sec latency 12275.876 ms 1 1278 0.00 MB/sec execute 3 sec latency 13276.089 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 1278 0.00 MB/sec execute 4 sec latency 14276.335 ms 1 1278 0.00 MB/sec execute 5 sec latency 15276.637 ms oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 1 1278 0.00 MB/sec execute 6 sec latency 16276.793 ms pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 1 1278 0.00 MB/sec execute 7 sec latency 17277.023 ms 1 1278 0.00 MB/sec execute 8 sec latency 18277.284 ms 1 1278 0.00 MB/sec execute 9 sec latency 19277.721 ms 1 1298 0.04 MB/sec execute 10 sec latency 20164.672 ms 1 1492 0.26 MB/sec execute 11 sec latency 73.785 ms oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1653 0.26 MB/sec execute 12 sec latency 41.350 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1805 0.25 MB/sec execute 13 sec latency 47.487 ms 1 1984 0.24 MB/sec execute 14 sec latency 73.833 ms 1 2158 0.24 MB/sec execute 15 sec latency 29.924 ms 1 2346 0.30 MB/sec execute 16 sec latency 89.391 ms 1 2429 0.29 MB/sec execute 17 sec latency 65.367 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3852 1283836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3748 1283940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 16588 3589928 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 56880 3549376 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 73468 7139304 2% /mnt/lustre 1 2569 0.32 MB/sec execute 18 sec latency 90.772 ms 1 2878 0.48 MB/sec execute 19 sec latency 30.646 ms 1 3026 0.52 MB/sec execute 20 sec latency 48.601 ms 1 3367 0.59 MB/sec execute 21 sec latency 34.121 ms 1 3626 0.70 MB/sec execute 22 sec latency 50.255 ms test_26 fail mds1 5 times Failing mds1 on oleg211-server 1 3767 0.72 MB/sec execute 23 sec latency 46.041 ms Stopping /mnt/lustre-mds1 (opts:) on oleg211-server 1 3861 0.69 MB/sec execute 24 sec latency 71.340 ms reboot facets: mds1 1 3990 0.68 MB/sec execute 25 sec latency 42.016 ms 1 4083 0.65 MB/sec execute 26 sec latency 351.295 ms 1 4083 0.63 MB/sec execute 27 sec latency 1351.573 ms 1 4083 0.61 MB/sec execute 28 sec latency 2351.790 ms 1 4083 0.58 MB/sec execute 29 sec latency 3352.052 ms 1 4083 0.56 MB/sec execute 30 sec latency 4352.287 ms 1 4083 0.55 MB/sec execute 31 sec latency 5352.534 ms 1 4083 0.53 MB/sec execute 32 sec latency 6352.777 ms 1 4083 0.51 MB/sec execute 33 sec latency 7353.023 ms 1 4083 0.50 MB/sec execute 34 sec latency 8353.254 ms 1 4083 0.48 MB/sec execute 35 sec latency 9353.435 ms Failover mds1 to oleg211-server mount facets: mds1 1 4083 0.47 MB/sec execute 36 sec latency 10353.707 ms 1 4083 0.46 MB/sec execute 37 sec latency 11353.927 ms 1 4083 0.45 MB/sec execute 38 sec latency 12354.312 ms 1 4083 0.43 MB/sec execute 39 sec latency 13354.654 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4083 0.42 MB/sec execute 40 sec latency 14355.117 ms 1 4083 0.41 MB/sec execute 41 sec latency 15355.364 ms oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 1 4083 0.40 MB/sec execute 42 sec latency 16355.732 ms pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4083 0.39 MB/sec execute 43 sec latency 17356.047 ms 1 4083 0.39 MB/sec execute 44 sec latency 18356.244 ms 1 4083 0.38 MB/sec execute 45 sec latency 19356.508 ms 1 4083 0.37 MB/sec execute 46 sec latency 20356.716 ms 1 4221 0.36 MB/sec execute 47 sec latency 20495.750 ms 1 4340 0.36 MB/sec execute 48 sec latency 68.171 ms oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4470 0.36 MB/sec execute 49 sec latency 42.473 ms 1 4619 0.38 MB/sec execute 50 sec latency 35.166 ms 1 4773 0.37 MB/sec execute 51 sec latency 32.404 ms 1 4960 0.42 MB/sec execute 52 sec latency 147.900 ms 1 5078 0.42 MB/sec execute 53 sec latency 107.494 ms 1 5216 0.41 MB/sec execute 54 sec latency 40.884 ms 1 5356 0.41 MB/sec execute 55 sec latency 66.851 ms 1 5543 0.40 MB/sec execute 56 sec latency 40.173 ms 1 5696 0.40 MB/sec execute 57 sec latency 50.518 ms 1 5861 0.41 MB/sec execute 58 sec latency 31.489 ms 1 5981 0.41 MB/sec execute 59 sec latency 75.929 ms 1 6129 0.42 MB/sec execute 60 sec latency 41.089 ms 1 6332 0.46 MB/sec execute 61 sec latency 78.252 ms 1 6552 0.48 MB/sec execute 62 sec latency 54.161 ms 1 6874 0.50 MB/sec execute 63 sec latency 41.020 ms 1 7109 0.54 MB/sec execute 64 sec latency 42.537 ms 1 7266 0.53 MB/sec execute 65 sec latency 27.571 ms 1 7404 0.54 MB/sec execute 66 sec latency 36.570 ms 1 7519 0.54 MB/sec execute 67 sec latency 76.692 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Oct 24 11:58:33 EDT 2025 with return code 0 69965 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 69965 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (223s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 11:58:37 (1761321517) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00437061 s, 937 kB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg211-server Stopping /mnt/lustre-ost1 (opts:) on oleg211-server reboot facets: ost1 Failover ost1 to oleg211-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00523392 s, 783 kB/s pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (37s) == replay-dual test 29: replay vs update with the same xid ========================================================== 11:59:14 (1761321554) SKIP: replay-dual test_29 needs >= 2 clients SKIP 29 (2s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 11:59:16 (1761321556) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0121455 s, 3.4 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.00807621 s, 5.1 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server reboot facets: mds1 Failover mds1 to oleg211-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 160+0 records in 160+0 records out 81920 bytes (82 kB, 80 KiB) copied, 24.9024 s, 3.3 kB/s oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (31s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 11:59:47 (1761321587) Failing ost1 on oleg211-server Stopping /mnt/lustre-ost1 (opts:) on oleg211-server striped dir -i1 -c2 -H crush /mnt/lustre/d31.replay-dual/lustre-OST0000 reboot facets: ost1 Creating to objid 2689 on ost lustre-OST0000... total: 32 open/close in 0.34 seconds: 93.59 ops/second at_max=0 fail_loc=0x80001420 Failover ost1 to oleg211-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE FULL state after 0 sec at_max=600 PASS 31 (32s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 12:00:19 (1761321619) fail_loc=0x0000131d fail_val=10 fail_loc=0x726 Stopping /mnt/lustre-mds2 (opts:) on oleg211-server Stopping /mnt/lustre-mds1 (opts:) on oleg211-server fail_loc=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2368 1285320 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1285580 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1412 3605604 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5404 3601420 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 6816 7207024 1% /mnt/lustre PASS 32 (27s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 12:00:46 (1761321646) at_min=60 Stopping /mnt/lustre-mds2 (opts:) on oleg211-server last_rcvd in /dev/mapper/mds2_flakey: incompat = 0x61c Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount REPLAY_WAIT mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in REPLAY_WAIT state after 0 sec oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds2 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 oleg211-server: *.lustre-MDT0001.recovery_status status: COMPLETE Stopping /mnt/lustre-mds2 (opts:) on oleg211-server last_rcvd in /dev/mapper/mds2_flakey: incompat = 0x61c Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 Starting client: oleg211-client.virtnet: -o user_xattr,flock oleg211-server@tcp:/lustre /mnt/lustre2 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in DISCONN FULL state after 0 sec affected facets: mds2 oleg211-server: oleg211-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 oleg211-server: *.lustre-MDT0001.recovery_status status: COMPLETE at_min=0 PASS 33 (47s) == replay-dual test complete, duration 3389 sec ========== 12:01:33 (1761321693) Stopping clients: oleg211-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg211-client.virtnet /mnt/lustre2 opts: