-----============= acceptance-small: replay-dual ============----- Fri Dec 5 07:16:10 EST 2025 oleg614-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b Starting client oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 Started clients oleg614-client.virtnet: 192.168.206.114@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) oleg614-client.virtnet: executing check_config_client /mnt/lustre oleg614-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg614-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93b1c52ef000.idle_timeout=debug osc.lustre-OST0000-osc-ffff93b1c53c5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93b1c52ef000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93b1c53c5800.idle_timeout=debug disable quota as required oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 osd-ldiskfs.track_declares_assert=1 == replay-dual test 0a: expired recovery with lost client ========================================================== 07:16:46 (1764937006) Check file is LU482_FAILED=/tmp/replay-dual.lu482.hcbayE UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 50 open/close in 1.02 seconds: 49.04 ops/second fail_loc=0x80000514 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1764937160 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second PASS 0a (160s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 07:19:26 (1764937166) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 PASS 0b (196s) == replay-dual test 1: |X| simple create ================= 07:22:42 (1764937362) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg614-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (57s) == replay-dual test 2: |X| mkdir adir ==================== 07:23:39 (1764937419) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (59s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 07:24:39 (1764937479) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (56s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 07:25:34 (1764937534) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1920 1285768 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1760 1285928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (55s) == replay-dual test 5: open, unlink |X| close ============ 07:26:29 (1764937589) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7741 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (59s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 07:27:28 (1764937648) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7741 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7741 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (55s) == replay-dual test 8: replay of resent request ========== 07:28:23 (1764937703) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (102s) == replay-dual test 9: resending a replayed create ======= 07:30:05 (1764937805) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (95s) == replay-dual test 10: resending a replayed unlink ====== 07:31:40 (1764937900) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (97s) == replay-dual test 11: both clients timeout during replay ========================================================== 07:33:17 (1764937997) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 31 sec fail_loc=0 PASS 11 (91s) == replay-dual test 12: open resend timeout ============== 07:34:48 (1764938088) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7741 fail_loc=0x80000302 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (91s) == replay-dual test 13: close resend timeout ============= 07:36:19 (1764938179) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7741 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (55s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 07:37:16 (1764938236) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.63 seconds: 39.88 ops/second total: 1 open/close in 0.03 seconds: 32.53 ops/second Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1764938383 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 PASS 15a (152s) == replay-dual test 15c: remove multiple OST orphans ===== 07:39:48 (1764938388) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 PASS 15c (221s) == replay-dual test 16: fail MDS during recovery (3571) == 07:43:29 (1764938609) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.62 seconds: 40.35 ops/second total: 1 open/close in 0.04 seconds: 22.77 ops/second Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1764938818 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 PASS 16 (214s) == replay-dual test 17: fail OST during recovery (3571) == 07:47:03 (1764938823) total: 25 open/close in 0.54 seconds: 46.16 ops/second total: 1 open/close in 0.04 seconds: 26.33 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing ost1 on oleg614-server Stopping /mnt/lustre-ost1 (opts:) on oleg614-server reboot facets: ost1 Failover ost1 to oleg614-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg614-server Stopping /mnt/lustre-ost1 (opts:) on oleg614-server reboot facets: ost1 Failover ost1 to oleg614-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-OST0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1764939018 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 PASS 17 (200s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 07:50:23 (1764939023) debug=+dlmtrace fail_loc=0x8000030b using seed 1719441247 running for 500 iterations total: 500 stats in 1 seconds: 500.000000 stats/second ldlm.namespaces.MGC192.168.206.114@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff93b1c3f74000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff93b1c4c29800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff93b1c3f74000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff93b1c4c29800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff93b1c3f74000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff93b1c4c29800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff93b1c3f74000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff93b1c4c29800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.206.114@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff93b1c3f74000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff93b1c4c29800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff93b1c3f74000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff93b1c4c29800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff93b1c3f74000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff93b1c4c29800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff93b1c3f74000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff93b1c4c29800.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (53s) == replay-dual test 19: resend of open request =========== 07:51:17 (1764939077) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1764939174.40 total 88.80 last 0.00) total: 1 open/close in 88.80 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (146s) == replay-dual test 20: recovery time is not increasing == 07:53:43 (1764939223) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 PASS 20 (399s) == replay-dual test 21a: commit on sharing =============== 08:00:21 (1764939621) mdt.lustre-MDT0000.commit_on_sharing=1 mdt.lustre-MDT0001.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 mdt.lustre-MDT0001.commit_on_sharing=0 PASS 21a (192s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 08:03:34 (1764939814) fail_loc=0x119 Failing mds2 on oleg614-server Stopping /mnt/lustre-mds2 (opts:) on oleg614-server reboot facets: mds2 Failover mds2 to oleg614-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1896 1285792 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.08 seconds: 26.30 ops/second total: 2 open/close in 0.04 seconds: 55.13 ops/second Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22a (86s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 08:05:00 (1764939900) fail_loc=0x119 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server Failing mds2 on oleg614-server Stopping /mnt/lustre-mds2 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg614-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.11 seconds: 17.86 ops/second total: 2 open/close in 0.03 seconds: 76.21 ops/second Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22b (108s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 08:06:48 (1764940008) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.05 seconds: 39.16 ops/second total: 2 open/close in 0.04 seconds: 47.75 ops/second Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22c (90s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 08:08:18 (1764940098) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server Failing mds2 on oleg614-server Stopping /mnt/lustre-mds2 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg614-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1712 1285976 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.06 seconds: 30.95 ops/second total: 2 open/close in 0.04 seconds: 53.95 ops/second Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22d (118s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 08:10:17 (1764940217) fail_loc=0x119 fail_loc=0 Failing mds2 on oleg614-server Stopping /mnt/lustre-mds2 (opts:) on oleg614-server reboot facets: mds2 Failover mds2 to oleg614-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 45.79 ops/second Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23a (72s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 08:11:29 (1764940289) fail_loc=0x119 fail_loc=0 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server Failing mds2 on oleg614-server Stopping /mnt/lustre-mds2 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg614-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 53.22 ops/second Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23b (99s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 08:13:08 (1764940388) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 50.43 ops/second Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23c (82s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 08:14:30 (1764940470) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server Failing mds2 on oleg614-server Stopping /mnt/lustre-mds2 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg614-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1708 1285980 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 57.55 ops/second Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23d (120s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 08:16:30 (1764940590) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (92s) == replay-dual test 25: replay|resend ==================== 08:18:02 (1764940682) 1+0 records in 1+0 records out 512 bytes copied, 0.00733132 s, 69.8 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg614-server Stopping /mnt/lustre-ost1 (opts:) on oleg614-server reboot facets: ost1 Failover ost1 to oleg614-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-OST0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6009: 65358 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (35s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 08:18:37 (1764940717) Starting client oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre Started clients oleg614-client.virtnet: 192.168.206.114@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Started tar loop with pid 66842 Started dbench loop with 66843 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Dec 5 08:18:40 EST 2025 waiting for dbench pid 66887 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1984 1285704 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1408 3605612 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3579820 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2796 7185432 1% /mnt/lustre 1 91 4.19 MB/sec warmup 1 sec latency 58.907 ms 1 157 3.27 MB/sec warmup 2 sec latency 104.069 ms 1 224 2.87 MB/sec warmup 3 sec latency 74.148 ms 1 305 2.80 MB/sec warmup 4 sec latency 68.324 ms 1 378 2.69 MB/sec warmup 5 sec latency 67.501 ms test_26 fail mds1 1 times 1 449 2.62 MB/sec warmup 6 sec latency 87.976 ms Failing mds1 on oleg614-server 1 521 2.58 MB/sec warmup 7 sec latency 69.258 ms Stopping /mnt/lustre-mds1 (opts:) on oleg614-server 1 661 2.64 MB/sec warmup 8 sec latency 113.860 ms 1 759 2.37 MB/sec warmup 9 sec latency 60.439 ms 1 865 2.16 MB/sec warmup 10 sec latency 45.817 ms 1 954 1.98 MB/sec warmup 11 sec latency 52.108 ms 1 995 1.82 MB/sec warmup 12 sec latency 419.899 ms 1 995 1.68 MB/sec warmup 13 sec latency 1420.109 ms 1 995 1.56 MB/sec warmup 14 sec latency 2420.281 ms reboot facets: mds1 1 995 1.46 MB/sec warmup 15 sec latency 3421.521 ms 1 995 1.37 MB/sec warmup 16 sec latency 4425.520 ms 1 995 1.29 MB/sec warmup 17 sec latency 5426.059 ms 1 995 1.22 MB/sec warmup 18 sec latency 6426.240 ms 1 995 1.15 MB/sec warmup 19 sec latency 7427.479 ms 1 995 0.00 MB/sec execute 1 sec latency 9429.602 ms 1 995 0.00 MB/sec execute 2 sec latency 10430.539 ms 1 995 0.00 MB/sec execute 3 sec latency 11430.734 ms 1 995 0.00 MB/sec execute 4 sec latency 12431.557 ms Failover mds1 to oleg614-server 1 995 0.00 MB/sec execute 5 sec latency 13432.508 ms mount facets: mds1 1 995 0.00 MB/sec execute 6 sec latency 14432.699 ms 1 995 0.00 MB/sec execute 7 sec latency 15432.830 ms 1 995 0.00 MB/sec execute 8 sec latency 16435.658 ms 1 995 0.00 MB/sec execute 9 sec latency 17435.856 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.00 MB/sec execute 10 sec latency 18436.054 ms 1 995 0.00 MB/sec execute 11 sec latency 19436.240 ms 1 995 0.00 MB/sec execute 12 sec latency 20436.575 ms 1 995 0.00 MB/sec execute 13 sec latency 21436.801 ms 1 995 0.00 MB/sec execute 14 sec latency 22437.596 ms 1 995 0.00 MB/sec execute 15 sec latency 23440.435 ms 1 995 0.00 MB/sec execute 16 sec latency 24441.595 ms 1 995 0.00 MB/sec execute 17 sec latency 25441.901 ms 1 995 0.00 MB/sec execute 18 sec latency 26442.091 ms 1 995 0.00 MB/sec execute 19 sec latency 27442.304 ms 1 995 0.00 MB/sec execute 20 sec latency 28442.556 ms oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 1 995 0.00 MB/sec execute 21 sec latency 29443.119 ms Started lustre-MDT0000 1 995 0.00 MB/sec execute 22 sec latency 30443.627 ms 1 995 0.00 MB/sec execute 23 sec latency 31444.664 ms 1 995 0.00 MB/sec execute 24 sec latency 32445.460 ms 1 1041 0.04 MB/sec execute 25 sec latency 33018.848 ms 1 1172 0.04 MB/sec execute 26 sec latency 48.120 ms oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1261 0.04 MB/sec execute 27 sec latency 45.329 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1447 0.15 MB/sec execute 28 sec latency 50.508 ms 1 1591 0.15 MB/sec execute 29 sec latency 40.590 ms 1 1698 0.14 MB/sec execute 30 sec latency 40.480 ms 1 1857 0.14 MB/sec execute 31 sec latency 35.107 ms 1 2018 0.14 MB/sec execute 32 sec latency 31.570 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1285380 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2136 1285552 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 22744 3583024 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 35920 3570464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 58664 7153488 1% /mnt/lustre 1 2158 0.15 MB/sec execute 33 sec latency 50.777 ms 1 2333 0.17 MB/sec execute 34 sec latency 33.601 ms 1 2426 0.18 MB/sec execute 35 sec latency 44.506 ms 1 2560 0.19 MB/sec execute 36 sec latency 50.967 ms 1 2763 0.27 MB/sec execute 37 sec latency 56.099 ms test_26 fail mds2 2 times 1 2973 0.31 MB/sec execute 38 sec latency 49.512 ms Failing mds2 on oleg614-server Stopping /mnt/lustre-mds2 (opts:) on oleg614-server 1 3209 0.32 MB/sec execute 39 sec latency 53.791 ms 1 3378 0.34 MB/sec execute 40 sec latency 45.546 ms reboot facets: mds2 1 3621 0.41 MB/sec execute 41 sec latency 35.244 ms 1 3760 0.42 MB/sec execute 42 sec latency 34.035 ms 1 3857 0.41 MB/sec execute 43 sec latency 41.694 ms 1 3952 0.41 MB/sec execute 44 sec latency 63.947 ms 1 4037 0.40 MB/sec execute 45 sec latency 65.202 ms 1 4083 0.39 MB/sec execute 46 sec latency 479.502 ms 1 4083 0.39 MB/sec execute 47 sec latency 1480.434 ms 1 4083 0.38 MB/sec execute 48 sec latency 2480.996 ms 1 4083 0.37 MB/sec execute 49 sec latency 3481.442 ms 1 4083 0.36 MB/sec execute 50 sec latency 4482.648 ms Failover mds2 to oleg614-server 1 4083 0.36 MB/sec execute 51 sec latency 5482.832 ms mount facets: mds2 1 4083 0.35 MB/sec execute 52 sec latency 6483.596 ms 1 4083 0.34 MB/sec execute 53 sec latency 7483.805 ms 1 4083 0.34 MB/sec execute 54 sec latency 8484.061 ms 1 4083 0.33 MB/sec execute 55 sec latency 9486.323 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4083 0.32 MB/sec execute 56 sec latency 10486.567 ms 1 4083 0.32 MB/sec execute 57 sec latency 11487.626 ms 1 4083 0.31 MB/sec execute 58 sec latency 12490.338 ms oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 1 4083 0.31 MB/sec execute 59 sec latency 13490.536 ms pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 1 4083 0.30 MB/sec execute 60 sec latency 14490.719 ms 1 4083 0.30 MB/sec execute 61 sec latency 15491.643 ms 1 4083 0.29 MB/sec execute 62 sec latency 16491.751 ms 1 4173 0.29 MB/sec execute 63 sec latency 16707.616 ms 1 4279 0.29 MB/sec execute 64 sec latency 66.920 ms oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4350 0.29 MB/sec execute 65 sec latency 67.554 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4443 0.29 MB/sec execute 66 sec latency 57.042 ms 1 4546 0.28 MB/sec execute 67 sec latency 44.904 ms 1 4693 0.30 MB/sec execute 68 sec latency 36.229 ms 1 4793 0.29 MB/sec execute 69 sec latency 50.471 ms 1 4971 0.33 MB/sec execute 70 sec latency 42.617 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2764 1284924 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24392 3579988 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 44292 3560632 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 68684 7140620 1% /mnt/lustre 1 5068 0.33 MB/sec execute 71 sec latency 56.022 ms 1 5170 0.32 MB/sec execute 72 sec latency 51.440 ms 1 5280 0.32 MB/sec execute 73 sec latency 48.691 ms 1 5370 0.32 MB/sec execute 74 sec latency 45.210 ms 1 5493 0.32 MB/sec execute 75 sec latency 40.990 ms 1 5627 0.31 MB/sec execute 76 sec latency 45.761 ms test_26 fail mds1 3 times 1 5759 0.31 MB/sec execute 77 sec latency 39.368 ms Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server 1 5912 0.32 MB/sec execute 78 sec latency 44.895 ms 1 5912 0.32 MB/sec execute 79 sec latency 1023.304 ms reboot facets: mds1 1 5912 0.31 MB/sec execute 80 sec latency 2024.067 ms 1 5912 0.31 MB/sec execute 81 sec latency 3025.295 ms 1 5912 0.31 MB/sec execute 82 sec latency 4025.503 ms 1 5912 0.30 MB/sec execute 83 sec latency 5025.779 ms 1 5912 0.30 MB/sec execute 84 sec latency 6026.811 ms 1 5912 0.30 MB/sec execute 85 sec latency 7027.009 ms 1 5912 0.29 MB/sec execute 86 sec latency 8027.326 ms 1 5912 0.29 MB/sec execute 87 sec latency 9027.630 ms 1 5912 0.29 MB/sec execute 88 sec latency 10027.826 ms 1 5912 0.28 MB/sec execute 89 sec latency 11029.316 ms Failover mds1 to oleg614-server 1 5912 0.28 MB/sec execute 90 sec latency 12029.493 ms mount facets: mds1 1 5912 0.28 MB/sec execute 91 sec latency 13029.680 ms 1 5912 0.27 MB/sec execute 92 sec latency 14030.712 ms 1 5912 0.27 MB/sec execute 93 sec latency 15030.861 ms 1 5912 0.27 MB/sec execute 94 sec latency 16031.369 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5912 0.26 MB/sec execute 95 sec latency 17033.074 ms 1 5912 0.26 MB/sec execute 96 sec latency 18033.255 ms 1 5912 0.26 MB/sec execute 97 sec latency 19033.576 ms oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 1 5912 0.26 MB/sec execute 98 sec latency 20034.021 ms Started lustre-MDT0000 1 5912 0.25 MB/sec execute 99 sec latency 21034.259 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 864 60.068 33018.834 Close 667 40.451 23607.361 Rename 33 31.627 53.766 Unlink 145 10.512 22.809 Qpathinfo 849 25.207 16707.594 Qfileinfo 142 0.838 8.127 Qfsinfo 122 0.856 10.740 Sfileinfo 70 18.443 54.652 Find 291 4.720 38.415 WriteX 424 5.497 35.790 ReadX 1257 0.466 66.417 LockX 2 4.266 4.635 UnlockX 2 4.151 5.330 Flush 50 26.224 56.452 Throughput 0.253432 MB/sec 1 clients 1 procs max_latency=33018.848 ms stopping dbench on /mnt/lustre at Fri Dec 5 08:20:43 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri Dec 5 08:20:47 EST 2025 waiting for dbench pid 70276 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 114 5.24 MB/sec warmup 1 sec latency 58.714 ms 1 197 3.92 MB/sec warmup 2 sec latency 59.700 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3268 1284420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3092 1284596 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37716 3553056 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11380 3594228 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49096 7147284 1% /mnt/lustre 1 279 3.48 MB/sec warmup 3 sec latency 63.559 ms 1 358 3.19 MB/sec warmup 4 sec latency 55.417 ms 1 446 3.12 MB/sec warmup 5 sec latency 83.164 ms 1 526 3.02 MB/sec warmup 6 sec latency 57.247 ms 1 689 3.02 MB/sec warmup 7 sec latency 43.533 ms test_26 fail mds2 4 times 1 789 2.68 MB/sec warmup 8 sec latency 42.775 ms Failing mds2 on oleg614-server 1 892 2.41 MB/sec warmup 9 sec latency 48.154 ms Stopping /mnt/lustre-mds2 (opts:) on oleg614-server 1 981 2.19 MB/sec warmup 10 sec latency 52.132 ms reboot facets: mds2 1 995 1.99 MB/sec warmup 11 sec latency 834.480 ms 1 995 1.82 MB/sec warmup 12 sec latency 1835.040 ms 1 995 1.68 MB/sec warmup 13 sec latency 2835.742 ms 1 995 1.56 MB/sec warmup 14 sec latency 3835.940 ms 1 995 1.46 MB/sec warmup 15 sec latency 4836.315 ms 1 995 1.37 MB/sec warmup 16 sec latency 5838.333 ms 1 995 1.29 MB/sec warmup 17 sec latency 6840.576 ms 1 995 1.22 MB/sec warmup 18 sec latency 7840.752 ms 1 995 1.15 MB/sec warmup 19 sec latency 8841.099 ms Failover mds2 to oleg614-server 1 995 0.00 MB/sec execute 1 sec latency 10842.861 ms mount facets: mds2 1 995 0.00 MB/sec execute 2 sec latency 11843.067 ms 1 995 0.00 MB/sec execute 3 sec latency 12843.227 ms 1 995 0.00 MB/sec execute 4 sec latency 13843.404 ms 1 995 0.00 MB/sec execute 5 sec latency 14844.356 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 995 0.00 MB/sec execute 6 sec latency 15844.620 ms 1 995 0.00 MB/sec execute 7 sec latency 16845.529 ms 1 995 0.00 MB/sec execute 8 sec latency 17845.715 ms oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 1 995 0.00 MB/sec execute 9 sec latency 18851.466 ms pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 1 995 0.00 MB/sec execute 10 sec latency 19852.524 ms 1 995 0.00 MB/sec execute 11 sec latency 20852.652 ms 1 995 0.00 MB/sec execute 12 sec latency 21853.617 ms 1 1026 0.07 MB/sec execute 13 sec latency 22454.731 ms 1 1169 0.08 MB/sec execute 14 sec latency 39.007 ms oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 1285 0.08 MB/sec execute 15 sec latency 38.332 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1492 0.26 MB/sec execute 16 sec latency 47.104 ms 1 1634 0.25 MB/sec execute 17 sec latency 41.460 ms 1 1773 0.24 MB/sec execute 18 sec latency 43.268 ms 1 1948 0.24 MB/sec execute 19 sec latency 33.433 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3696 1283992 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3644 1284044 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 50648 3555976 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 22484 3584080 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 73132 7140056 2% /mnt/lustre 1 2097 0.24 MB/sec execute 20 sec latency 33.602 ms 1 2290 0.28 MB/sec execute 21 sec latency 29.653 ms 1 2417 0.28 MB/sec execute 22 sec latency 39.452 ms 1 2559 0.30 MB/sec execute 23 sec latency 40.415 ms 1 2785 0.42 MB/sec execute 24 sec latency 38.244 ms 1 2997 0.46 MB/sec execute 25 sec latency 45.016 ms test_26 fail mds1 5 times Failing mds1 on oleg614-server 1 3275 0.52 MB/sec execute 26 sec latency 46.083 ms Stopping /mnt/lustre-mds1 (opts:) on oleg614-server 1 3399 0.50 MB/sec execute 27 sec latency 76.770 ms 1 3627 0.60 MB/sec execute 28 sec latency 38.133 ms reboot facets: mds1 1 3760 0.62 MB/sec execute 29 sec latency 35.007 ms 1 3851 0.59 MB/sec execute 30 sec latency 56.272 ms 1 3945 0.59 MB/sec execute 31 sec latency 65.930 ms 1 4049 0.57 MB/sec execute 32 sec latency 45.321 ms 1 4083 0.55 MB/sec execute 33 sec latency 735.743 ms 1 4083 0.53 MB/sec execute 34 sec latency 1735.909 ms 1 4083 0.52 MB/sec execute 35 sec latency 2736.153 ms 1 4083 0.50 MB/sec execute 36 sec latency 3736.343 ms 1 4083 0.49 MB/sec execute 37 sec latency 4736.601 ms 1 4083 0.48 MB/sec execute 38 sec latency 5737.727 ms Failover mds1 to oleg614-server 1 4083 0.47 MB/sec execute 39 sec latency 6739.912 ms mount facets: mds1 1 4083 0.45 MB/sec execute 40 sec latency 7740.592 ms 1 4083 0.44 MB/sec execute 41 sec latency 8740.858 ms 1 4083 0.43 MB/sec execute 42 sec latency 9741.038 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4083 0.42 MB/sec execute 43 sec latency 10741.259 ms 1 4083 0.41 MB/sec execute 44 sec latency 11741.419 ms 1 4083 0.40 MB/sec execute 45 sec latency 12741.631 ms 1 4083 0.39 MB/sec execute 46 sec latency 13741.874 ms 1 4083 0.39 MB/sec execute 47 sec latency 14742.332 ms oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 1 4083 0.38 MB/sec execute 48 sec latency 15742.877 ms pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4083 0.37 MB/sec execute 49 sec latency 16744.277 ms 1 4083 0.36 MB/sec execute 50 sec latency 17744.455 ms 1 4083 0.36 MB/sec execute 51 sec latency 18745.058 ms 1 4083 0.35 MB/sec execute 52 sec latency 19745.303 ms 1 4214 0.34 MB/sec execute 53 sec latency 19880.996 ms 1 4309 0.34 MB/sec execute 54 sec latency 69.592 ms oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4427 0.34 MB/sec execute 55 sec latency 45.641 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4503 0.34 MB/sec execute 56 sec latency 85.123 ms 1 4647 0.35 MB/sec execute 57 sec latency 35.926 ms kill: sending signal to 66865 failed: No such process 1 4780 0.35 MB/sec execute 58 sec latency 37.902 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri Dec 5 08:22:05 EST 2025 with return code 0 70276 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 70276 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (235s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 08:22:32 (1764940952) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00432712 s, 947 kB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg614-server Stopping /mnt/lustre-ost1 (opts:) on oleg614-server reboot facets: ost1 Failover ost1 to oleg614-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00630146 s, 650 kB/s oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-OST0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (32s) == replay-dual test 29: replay vs update with the same xid ========================================================== 08:23:04 (1764940984) SKIP: replay-dual test_29 needs >= 2 clients SKIP 29 (2s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 08:23:06 (1764940986) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0118084 s, 3.5 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.00769722 s, 5.3 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server reboot facets: mds1 Failover mds1 to oleg614-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 160+0 records in 160+0 records out 81920 bytes (82 kB, 80 KiB) copied, 24.3301 s, 3.4 kB/s oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (31s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 08:23:37 (1764941017) Failing ost1 on oleg614-server Stopping /mnt/lustre-ost1 (opts:) on oleg614-server striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d31.replay-dual/lustre-OST0000 reboot facets: ost1 Creating to objid 2593 on ost lustre-OST0000... total: 32 open/close in 0.33 seconds: 96.89 ops/second at_max=0 fail_loc=0x80001420 Failover ost1 to oleg614-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-OST0000 oleg614-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL IDLE state after 0 sec at_max=600 PASS 31 (33s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 08:24:10 (1764941050) fail_loc=0x0000131d fail_val=10 fail_loc=0x726 Stopping /mnt/lustre-mds2 (opts:) on oleg614-server Stopping /mnt/lustre-mds1 (opts:) on oleg614-server fail_loc=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0001 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1285280 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2136 1285552 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1412 3605604 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605628 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2800 7211232 1% /mnt/lustre PASS 32 (27s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 08:24:37 (1764941077) Stopping /mnt/lustre-mds1 (opts:) on oleg614-server last_rcvd in /dev/mapper/mds1_flakey: incompat = 0x61c Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg614-client.virtnet: executing wait_import_state_mount (REPLAY_WAIT|REPLAY_LOCKS) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in REPLAY_LOCKS state after 0 sec oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping /mnt/lustre-mds1 (opts:) on oleg614-server last_rcvd in /dev/mapper/mds1_flakey: incompat = 0x61c Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg614-server: oleg614-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg614-client: oleg614-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg614-client.virtnet: -o user_xattr,flock oleg614-server@tcp:/lustre /mnt/lustre2 mount.lustre: mount oleg614-server@tcp:/lustre at /mnt/lustre2 failed: Input/output error Is the MGS running? oleg614-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec PASS 33 (50s) == replay-dual test complete, duration 4156 sec ========== 08:25:27 (1764941127) Stopping clients: oleg614-client.virtnet /mnt/lustre2 (opts:)