-----============= acceptance-small: replay-dual ============----- Fri May 16 08:45:38 EDT 2025 oleg349-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex: No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b Starting client oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 Started clients oleg349-client.virtnet: 192.168.203.149@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) oleg349-client.virtnet: executing check_config_client /mnt/lustre oleg349-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg349-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a9864000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b6328000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9864000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6328000.idle_timeout=debug disable quota as required oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 osd-ldiskfs.track_declares_assert=1 == replay-dual test 0a: expired recovery with lost client ========================================================== 08:46:03 (1747399563) Check file is LU482_FAILED=/tmp/replay-dual.lu482.KTrsMi UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 50 open/close in 0.66 seconds: 75.46 ops/second fail_loc=0x80000514 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1747399679 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second PASS 0a (119s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 08:48:02 (1747399682) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 PASS 0b (133s) == replay-dual test 1: |X| simple create ================= 08:50:15 (1747399815) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg349-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (35s) == replay-dual test 2: |X| mkdir adir ==================== 08:50:51 (1747399851) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (43s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 08:51:33 (1747399893) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (33s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 08:52:06 (1747399926) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1760 1285928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (33s) == replay-dual test 5: open, unlink |X| close ============ 08:52:39 (1747399959) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7503 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (39s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 08:53:18 (1747399998) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7503 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7503 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (39s) == replay-dual test 8: replay of resent request ========== 08:53:57 (1747400037) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (49s) == replay-dual test 9: resending a replayed create ======= 08:54:46 (1747400086) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (48s) == replay-dual test 10: resending a replayed unlink ====== 08:55:34 (1747400134) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (48s) == replay-dual test 11: both clients timeout during replay ========================================================== 08:56:22 (1747400182) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec fail_loc=0 PASS 11 (51s) == replay-dual test 12: open resend timeout ============== 08:57:13 (1747400233) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7503 fail_loc=0x80000302 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (27s) == replay-dual test 13: close resend timeout ============= 08:57:40 (1747400260) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7503 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (28s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 08:58:09 (1747400289) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.20 seconds: 127.88 ops/second total: 1 open/close in 0.01 seconds: 115.01 ops/second Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1747400379 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 PASS 15a (91s) == replay-dual test 15c: remove multiple OST orphans ===== 08:59:40 (1747400380) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 PASS 15c (112s) == replay-dual test 16: fail MDS during recovery (3571) == 09:01:32 (1747400492) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.15 seconds: 166.82 ops/second total: 1 open/close in 0.01 seconds: 165.04 ops/second Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1747400621 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 PASS 16 (130s) == replay-dual test 17: fail OST during recovery (3571) == 09:03:42 (1747400622) total: 25 open/close in 0.20 seconds: 125.68 ops/second total: 1 open/close in 0.01 seconds: 94.01 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing ost1 on oleg349-server Stopping /mnt/lustre-ost1 (opts:) on oleg349-server reboot facets: ost1 Failover ost1 to oleg349-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg349-server Stopping /mnt/lustre-ost1 (opts:) on oleg349-server reboot facets: ost1 Failover ost1 to oleg349-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1747400745 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 PASS 17 (125s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 09:05:47 (1747400747) debug=+dlmtrace using seed 382122341 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.203.149@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800aa6a3000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012e285000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800aa6a3000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff88012e285000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800aa6a3000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff88012e285000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800aa6a3000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff88012e285000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.203.149@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800aa6a3000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012e285000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800aa6a3000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff88012e285000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800aa6a3000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff88012e285000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800aa6a3000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012e285000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (45s) == replay-dual test 19: resend of open request =========== 09:06:32 (1747400792) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1747400870.90 total 76.02 last 0.00) total: 1 open/close in 76.02 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (107s) == replay-dual test 20: recovery time is not increasing == 09:08:19 (1747400899) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 PASS 20 (317s) == replay-dual test 21a: commit on sharing =============== 09:13:36 (1747401216) mdt.lustre-MDT0000.commit_on_sharing=1 mdt.lustre-MDT0001.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 mdt.lustre-MDT0001.commit_on_sharing=0 PASS 21a (157s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 09:16:13 (1747401373) fail_loc=0x119 Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server reboot facets: mds2 Failover mds2 to oleg349-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1896 1285792 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 89.12 ops/second total: 2 open/close in 0.01 seconds: 176.15 ops/second Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22a (52s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 09:17:05 (1747401425) fail_loc=0x119 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg349-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 80.74 ops/second total: 2 open/close in 0.01 seconds: 162.40 ops/second Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22b (88s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 09:18:33 (1747401513) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1285796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1748 1285940 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 116.58 ops/second total: 2 open/close in 0.01 seconds: 222.00 ops/second Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22c (57s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 09:19:30 (1747401570) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg349-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1856 1285832 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1712 1285976 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 100.38 ops/second total: 2 open/close in 0.01 seconds: 219.21 ops/second Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22d (86s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 09:20:56 (1747401656) fail_loc=0x119 fail_loc=0 Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server reboot facets: mds2 Failover mds2 to oleg349-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.01 seconds: 159.48 ops/second Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23a (52s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 09:21:48 (1747401708) fail_loc=0x119 fail_loc=0 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg349-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.01 seconds: 164.61 ops/second Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23b (71s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 09:22:59 (1747401779) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1285800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1744 1285944 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 125.45 ops/second Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23c (56s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 09:23:55 (1747401835) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to oleg349-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1285836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1708 1285980 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre total: 2 open/close in 0.01 seconds: 160.91 ops/second Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23d (84s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 09:25:19 (1747401919) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (78s) == replay-dual test 25: replay|resend ==================== 09:26:37 (1747401997) 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00261029 s, 196 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg349-server Stopping /mnt/lustre-ost1 (opts:) on oleg349-server reboot facets: ost1 Failover ost1 to oleg349-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5990: 5184 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (21s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 09:26:58 (1747402018) Starting client oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre Started clients oleg349-client.virtnet: 192.168.203.149@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Started tar loop with pid 6790 Started dbench loop with 6791 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri May 16 09:26:59 EDT 2025 waiting for dbench pid 6826 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 287 10.70 MB/sec warmup 1 sec latency 17.205 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2268 1285420 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1285628 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25984 3556088 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27372 7161720 1% /mnt/lustre 1 580 9.85 MB/sec warmup 2 sec latency 20.676 ms 1 796 7.16 MB/sec warmup 3 sec latency 440.695 ms 1 1170 5.75 MB/sec warmup 4 sec latency 13.731 ms test_26 fail mds1 1 times 1 1614 5.23 MB/sec warmup 5 sec latency 14.224 ms Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server 1 1877 4.39 MB/sec warmup 6 sec latency 264.320 ms 1 1877 3.77 MB/sec warmup 7 sec latency 1264.483 ms 1 1877 3.30 MB/sec warmup 8 sec latency 2264.645 ms 1 1877 2.93 MB/sec warmup 9 sec latency 3264.838 ms 1 1877 2.64 MB/sec warmup 10 sec latency 4264.999 ms 1 1877 2.40 MB/sec warmup 11 sec latency 5265.123 ms 1 1877 2.20 MB/sec warmup 12 sec latency 6265.299 ms reboot facets: mds1 1 1877 2.03 MB/sec warmup 13 sec latency 7265.479 ms 1 1877 1.88 MB/sec warmup 14 sec latency 8265.652 ms 1 1877 1.76 MB/sec warmup 15 sec latency 9265.843 ms 1 1877 1.65 MB/sec warmup 16 sec latency 10265.981 ms 1 1877 1.55 MB/sec warmup 17 sec latency 11266.113 ms 1 1877 1.46 MB/sec warmup 18 sec latency 12266.244 ms 1 1877 1.39 MB/sec warmup 19 sec latency 13266.477 ms 1 1877 0.00 MB/sec execute 1 sec latency 15266.778 ms 1 1877 0.00 MB/sec execute 2 sec latency 16266.947 ms Failover mds1 to oleg349-server mount facets: mds1 1 1877 0.00 MB/sec execute 3 sec latency 17267.081 ms 1 1877 0.00 MB/sec execute 4 sec latency 18267.223 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1877 0.00 MB/sec execute 5 sec latency 19267.387 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 1 1877 0.00 MB/sec execute 6 sec latency 20267.569 ms pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 1 1877 0.00 MB/sec execute 7 sec latency 21267.779 ms 1 1877 0.00 MB/sec execute 8 sec latency 22268.008 ms 1 1877 0.00 MB/sec execute 9 sec latency 23268.152 ms 1 1877 0.00 MB/sec execute 10 sec latency 24268.306 ms 1 1973 0.01 MB/sec execute 11 sec latency 25073.540 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2376 0.13 MB/sec execute 12 sec latency 13.664 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2919 0.55 MB/sec execute 13 sec latency 15.137 ms 1 3587 0.86 MB/sec execute 14 sec latency 16.091 ms 1 3939 0.91 MB/sec execute 15 sec latency 12.586 ms 1 4231 0.87 MB/sec execute 16 sec latency 16.298 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2820 1284868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36916 3566340 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12768 3590876 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49684 7157216 1% /mnt/lustre 1 4550 0.86 MB/sec execute 17 sec latency 19.677 ms 1 5019 1.04 MB/sec execute 18 sec latency 15.202 ms 1 5379 1.00 MB/sec execute 19 sec latency 12.958 ms test_26 fail mds2 2 times 1 5760 0.98 MB/sec execute 20 sec latency 14.643 ms Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server 1 6012 0.99 MB/sec execute 21 sec latency 156.715 ms reboot facets: mds2 1 6012 0.95 MB/sec execute 22 sec latency 1156.950 ms 1 6012 0.91 MB/sec execute 23 sec latency 2157.195 ms 1 6012 0.87 MB/sec execute 24 sec latency 3157.414 ms 1 6012 0.83 MB/sec execute 25 sec latency 4157.556 ms 1 6012 0.80 MB/sec execute 26 sec latency 5157.744 ms 1 6012 0.77 MB/sec execute 27 sec latency 6157.973 ms 1 6012 0.75 MB/sec execute 28 sec latency 7158.186 ms 1 6012 0.72 MB/sec execute 29 sec latency 8158.343 ms 1 6012 0.70 MB/sec execute 30 sec latency 9158.538 ms 1 6012 0.67 MB/sec execute 31 sec latency 10158.692 ms Failover mds2 to oleg349-server mount facets: mds2 1 6012 0.65 MB/sec execute 32 sec latency 11158.809 ms 1 6012 0.63 MB/sec execute 33 sec latency 12158.908 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 6012 0.61 MB/sec execute 34 sec latency 13159.014 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 1 6012 0.60 MB/sec execute 35 sec latency 14159.138 ms pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 1 6012 0.58 MB/sec execute 36 sec latency 15159.244 ms 1 6012 0.56 MB/sec execute 37 sec latency 16159.357 ms 1 6012 0.55 MB/sec execute 38 sec latency 17159.508 ms 1 6012 0.53 MB/sec execute 39 sec latency 18159.683 ms 1 6012 0.52 MB/sec execute 40 sec latency 19159.816 ms 1 6049 0.52 MB/sec execute 41 sec latency 20030.579 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 6724 0.64 MB/sec execute 42 sec latency 20.813 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7230 0.73 MB/sec execute 43 sec latency 12.989 ms 1 7516 0.75 MB/sec execute 44 sec latency 14.901 ms 1 7856 0.74 MB/sec execute 45 sec latency 13.308 ms 1 8237 0.76 MB/sec execute 46 sec latency 13.415 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2808 1284880 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3556 1284132 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37332 3556040 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13388 3585528 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50720 7141568 1% /mnt/lustre 1 8633 0.81 MB/sec execute 47 sec latency 42.864 ms 1 8887 0.79 MB/sec execute 48 sec latency 322.913 ms 1 9340 0.79 MB/sec execute 49 sec latency 9.179 ms test_26 fail mds1 3 times 1 9724 0.82 MB/sec execute 50 sec latency 36.612 ms Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server 1 10463 0.93 MB/sec execute 51 sec latency 15.935 ms reboot facets: mds1 1 10910 0.99 MB/sec execute 52 sec latency 13.774 ms 1 11157 0.98 MB/sec execute 53 sec latency 14.843 ms 1 11174 0.96 MB/sec execute 54 sec latency 950.072 ms 1 11174 0.94 MB/sec execute 55 sec latency 1950.179 ms 1 11174 0.93 MB/sec execute 56 sec latency 2950.289 ms 1 11174 0.91 MB/sec execute 57 sec latency 3950.406 ms 1 11174 0.90 MB/sec execute 58 sec latency 4950.547 ms 1 11174 0.88 MB/sec execute 59 sec latency 5950.706 ms 1 11174 0.87 MB/sec execute 60 sec latency 6950.825 ms 1 11174 0.85 MB/sec execute 61 sec latency 7950.945 ms Failover mds1 to oleg349-server mount facets: mds1 1 11174 0.84 MB/sec execute 62 sec latency 8951.072 ms 1 11174 0.82 MB/sec execute 63 sec latency 9951.250 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 11174 0.81 MB/sec execute 64 sec latency 10951.395 ms 1 11174 0.80 MB/sec execute 65 sec latency 11951.521 ms 1 11174 0.79 MB/sec execute 66 sec latency 12951.644 ms 1 11174 0.78 MB/sec execute 67 sec latency 13951.764 ms 1 11174 0.76 MB/sec execute 68 sec latency 14951.924 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 1 11174 0.75 MB/sec execute 69 sec latency 15952.050 ms pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 1 11174 0.74 MB/sec execute 70 sec latency 16952.165 ms 1 11174 0.73 MB/sec execute 71 sec latency 17952.301 ms 1 11174 0.72 MB/sec execute 72 sec latency 18952.437 ms 1 11174 0.71 MB/sec execute 73 sec latency 19952.592 ms 1 11480 0.71 MB/sec execute 74 sec latency 20050.424 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11820 0.72 MB/sec execute 75 sec latency 16.005 ms 1 12239 0.75 MB/sec execute 76 sec latency 13.350 ms 1 12612 0.75 MB/sec execute 77 sec latency 23.972 ms 1 13032 0.76 MB/sec execute 78 sec latency 11.878 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3048 1284640 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4692 1282996 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 45696 3551916 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24128 3570776 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 69824 7122692 1% /mnt/lustre 1 13546 0.81 MB/sec execute 79 sec latency 18.045 ms 1 13887 0.82 MB/sec execute 80 sec latency 416.012 ms 1 14364 0.86 MB/sec execute 81 sec latency 13.209 ms 1 14634 0.87 MB/sec execute 82 sec latency 14.177 ms test_26 fail mds2 4 times Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server 1 14976 0.86 MB/sec execute 83 sec latency 14.481 ms reboot facets: mds2 1 15200 0.86 MB/sec execute 84 sec latency 251.991 ms 1 15200 0.85 MB/sec execute 85 sec latency 1252.154 ms 1 15200 0.84 MB/sec execute 86 sec latency 2252.286 ms 1 15200 0.83 MB/sec execute 87 sec latency 3252.440 ms 1 15200 0.82 MB/sec execute 88 sec latency 4252.577 ms 1 15200 0.81 MB/sec execute 89 sec latency 5252.739 ms 1 15200 0.80 MB/sec execute 90 sec latency 6252.902 ms 1 15200 0.79 MB/sec execute 91 sec latency 7253.044 ms 1 15200 0.78 MB/sec execute 92 sec latency 8253.189 ms 1 15200 0.77 MB/sec execute 93 sec latency 9253.333 ms Failover mds2 to oleg349-server 1 15200 0.77 MB/sec execute 94 sec latency 10253.422 ms mount facets: mds2 1 15200 0.76 MB/sec execute 95 sec latency 11253.511 ms 1 15200 0.75 MB/sec execute 96 sec latency 12253.631 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 15200 0.74 MB/sec execute 97 sec latency 13253.744 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 1 15200 0.73 MB/sec execute 98 sec latency 14253.859 ms 1 15200 0.73 MB/sec execute 99 sec latency 15253.986 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 0 cleanup 103 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 2278 26.574 25073.489 Close 1670 1.369 8.210 Rename 97 11.042 42.841 Unlink 482 45.284 20030.562 Qpathinfo 2087 11.486 20050.409 Qfileinfo 357 0.350 1.823 Qfsinfo 392 0.694 14.632 Sfileinfo 184 5.148 10.286 Find 813 1.416 18.792 WriteX 1140 1.902 8.932 ReadX 3639 0.032 2.256 LockX 8 1.239 1.418 UnlockX 8 1.282 1.397 Flush 169 11.693 416.001 Throughput 0.727447 MB/sec 1 clients 1 procs max_latency=25073.540 ms stopping dbench on /mnt/lustre at Fri May 16 09:29:02 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Fri May 16 09:29:03 EDT 2025 waiting for dbench pid 11600 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 292 10.70 MB/sec warmup 1 sec latency 18.238 ms 1 601 10.48 MB/sec warmup 2 sec latency 28.525 ms 1 948 7.27 MB/sec warmup 3 sec latency 17.026 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3572 1284116 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 5228 1282460 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 36992 3529232 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18104 3568116 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 55096 7097348 1% /mnt/lustre 1 1281 5.79 MB/sec warmup 4 sec latency 15.835 ms 1 1660 5.24 MB/sec warmup 5 sec latency 178.307 ms 1 2093 4.44 MB/sec warmup 6 sec latency 13.037 ms 1 2490 4.10 MB/sec warmup 7 sec latency 25.861 ms test_26 fail mds1 5 times Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server 1 3197 4.29 MB/sec warmup 8 sec latency 13.410 ms 1 3720 4.31 MB/sec warmup 9 sec latency 31.282 ms 1 4010 4.01 MB/sec warmup 10 sec latency 14.043 ms 1 4083 3.64 MB/sec warmup 11 sec latency 753.000 ms 1 4083 3.34 MB/sec warmup 12 sec latency 1753.156 ms 1 4083 3.08 MB/sec warmup 13 sec latency 2753.319 ms reboot facets: mds1 1 4083 2.86 MB/sec warmup 14 sec latency 3753.483 ms 1 4083 2.67 MB/sec warmup 15 sec latency 4753.703 ms 1 4083 2.51 MB/sec warmup 16 sec latency 5753.856 ms 1 4083 2.36 MB/sec warmup 17 sec latency 6754.018 ms 1 4083 2.23 MB/sec warmup 18 sec latency 7754.210 ms 1 4083 2.11 MB/sec warmup 19 sec latency 8754.378 ms 1 4083 0.00 MB/sec execute 1 sec latency 10754.712 ms 1 4083 0.00 MB/sec execute 2 sec latency 11754.851 ms 1 4083 0.00 MB/sec execute 3 sec latency 12754.995 ms Failover mds1 to oleg349-server 1 4083 0.00 MB/sec execute 4 sec latency 13755.156 ms mount facets: mds1 1 4083 0.00 MB/sec execute 5 sec latency 14755.320 ms 1 4083 0.00 MB/sec execute 6 sec latency 15755.439 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 4083 0.00 MB/sec execute 7 sec latency 16755.637 ms 1 4083 0.00 MB/sec execute 8 sec latency 17755.790 ms 1 4083 0.00 MB/sec execute 9 sec latency 18755.949 ms 1 4083 0.00 MB/sec execute 10 sec latency 19756.113 ms 1 4083 0.00 MB/sec execute 11 sec latency 20756.275 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 1 4083 0.00 MB/sec execute 12 sec latency 21756.469 ms 1 4083 0.00 MB/sec execute 13 sec latency 22756.594 ms 1 4083 0.00 MB/sec execute 14 sec latency 23756.728 ms 1 4083 0.00 MB/sec execute 15 sec latency 24756.886 ms 1 4083 0.00 MB/sec execute 16 sec latency 25757.009 ms 1 4367 0.03 MB/sec execute 17 sec latency 25814.113 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4687 0.11 MB/sec execute 18 sec latency 17.214 ms 1 5093 0.27 MB/sec execute 19 sec latency 17.855 ms 1 5422 0.27 MB/sec execute 20 sec latency 13.676 ms 1 5893 0.33 MB/sec execute 21 sec latency 9.607 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2692 1284996 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3728 1283960 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 45788 3559316 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 21000 3585360 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 66788 7144676 1% /mnt/lustre 1 6167 0.37 MB/sec execute 22 sec latency 59.611 ms 1 6555 0.55 MB/sec execute 23 sec latency 368.394 ms 1 7164 0.73 MB/sec execute 24 sec latency 402.072 ms 1 7447 0.76 MB/sec execute 25 sec latency 14.049 ms test_26 fail mds2 6 times Failing mds2 on oleg349-server Stopping /mnt/lustre-mds2 (opts:) on oleg349-server 1 7722 0.74 MB/sec execute 26 sec latency 16.200 ms reboot facets: mds2 1 8026 0.74 MB/sec execute 27 sec latency 66.359 ms 1 8106 0.72 MB/sec execute 28 sec latency 717.813 ms 1 8106 0.69 MB/sec execute 29 sec latency 1718.013 ms 1 8106 0.67 MB/sec execute 30 sec latency 2718.179 ms 1 8106 0.65 MB/sec execute 31 sec latency 3718.349 ms 1 8106 0.63 MB/sec execute 32 sec latency 4718.512 ms 1 8106 0.61 MB/sec execute 33 sec latency 5718.667 ms 1 8106 0.59 MB/sec execute 34 sec latency 6718.833 ms 1 8106 0.57 MB/sec execute 35 sec latency 7718.980 ms 1 8106 0.56 MB/sec execute 36 sec latency 8719.162 ms Failover mds2 to oleg349-server 1 8106 0.54 MB/sec execute 37 sec latency 9719.326 ms mount facets: mds2 1 8106 0.53 MB/sec execute 38 sec latency 10719.450 ms 1 8106 0.51 MB/sec execute 39 sec latency 11719.570 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 8106 0.50 MB/sec execute 40 sec latency 12719.697 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 1 8106 0.49 MB/sec execute 41 sec latency 13719.860 ms Started lustre-MDT0001 1 8106 0.48 MB/sec execute 42 sec latency 14720.009 ms 1 8106 0.47 MB/sec execute 43 sec latency 15720.158 ms 1 8106 0.46 MB/sec execute 44 sec latency 16720.332 ms 1 8106 0.45 MB/sec execute 45 sec latency 17720.460 ms 1 8144 0.46 MB/sec execute 46 sec latency 18601.140 ms 1 8519 0.51 MB/sec execute 47 sec latency 19.413 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8797 0.51 MB/sec execute 48 sec latency 18.065 ms 1 9140 0.50 MB/sec execute 49 sec latency 14.547 ms 1 9493 0.52 MB/sec execute 50 sec latency 49.562 ms 1 9995 0.61 MB/sec execute 51 sec latency 15.484 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2508 1285180 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 43160 3561604 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 20124 3583980 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 63284 7145584 1% /mnt/lustre 1 10476 0.65 MB/sec execute 52 sec latency 14.651 ms 1 10879 0.72 MB/sec execute 53 sec latency 17.366 ms 1 11098 0.71 MB/sec execute 54 sec latency 27.334 ms 1 11419 0.70 MB/sec execute 55 sec latency 24.808 ms test_26 fail mds1 7 times striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server 1 11718 0.72 MB/sec execute 56 sec latency 23.920 ms reboot facets: mds1 1 11769 0.71 MB/sec execute 57 sec latency 878.030 ms 1 11769 0.69 MB/sec execute 58 sec latency 1878.250 ms 1 11769 0.68 MB/sec execute 59 sec latency 2878.469 ms 1 11769 0.67 MB/sec execute 60 sec latency 3878.680 ms 1 11769 0.66 MB/sec execute 61 sec latency 4878.919 ms 1 11769 0.65 MB/sec execute 62 sec latency 5879.133 ms 1 11769 0.64 MB/sec execute 63 sec latency 6879.353 ms 1 11769 0.63 MB/sec execute 64 sec latency 7879.595 ms 1 11769 0.62 MB/sec execute 65 sec latency 8879.812 ms 1 11769 0.61 MB/sec execute 66 sec latency 9880.070 ms Failover mds1 to oleg349-server 1 11769 0.60 MB/sec execute 67 sec latency 10880.217 ms mount facets: mds1 1 11769 0.59 MB/sec execute 68 sec latency 11880.398 ms 1 11769 0.58 MB/sec execute 69 sec latency 12880.553 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 11769 0.57 MB/sec execute 70 sec latency 13880.701 ms 1 11769 0.57 MB/sec execute 71 sec latency 14880.887 ms 1 11769 0.56 MB/sec execute 72 sec latency 15881.028 ms 1 11769 0.55 MB/sec execute 73 sec latency 16881.199 ms oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 1 11769 0.54 MB/sec execute 74 sec latency 17881.420 ms Started lustre-MDT0000 1 11769 0.54 MB/sec execute 75 sec latency 18881.579 ms 1 11769 0.53 MB/sec execute 76 sec latency 19881.736 ms 1 11769 0.52 MB/sec execute 77 sec latency 20881.896 ms 1 11769 0.52 MB/sec execute 78 sec latency 21882.050 ms 1 12059 0.55 MB/sec execute 79 sec latency 22215.855 ms oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 12380 0.54 MB/sec execute 80 sec latency 14.851 ms 1 12803 0.54 MB/sec execute 81 sec latency 11.347 ms 1 13214 0.56 MB/sec execute 82 sec latency 12.981 ms 1 13896 0.63 MB/sec execute 83 sec latency 13.428 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Fri May 16 09:30:47 EDT 2025 with return code 0 11600 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 11600 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (233s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 09:30:51 (1747402251) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00245976 s, 1.7 MB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg349-server Stopping /mnt/lustre-ost1 (opts:) on oleg349-server reboot facets: ost1 Failover ost1 to oleg349-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00277721 s, 1.5 MB/s Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (30s) == replay-dual test 29: replay vs update with the same xid ========================================================== 09:31:21 (1747402281) SKIP: replay-dual test_29 needs >= 2 clients SKIP 29 (0s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 09:31:21 (1747402281) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0064932 s, 6.3 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.00450977 s, 9.1 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server reboot facets: mds1 Failover mds1 to oleg349-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 160+0 records in 160+0 records out 81920 bytes (82 kB) copied, 23.8542 s, 3.4 kB/s oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (28s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 09:31:49 (1747402309) Failing ost1 on oleg349-server Stopping /mnt/lustre-ost1 (opts:) on oleg349-server reboot facets: ost1 striped dir -i1 -c2 -H crush /mnt/lustre/d31.replay-dual/lustre-OST0000 Creating to objid 2849 on ost lustre-OST0000... total: 32 open/close in 0.13 seconds: 243.11 ops/second at_max=0 fail_loc=0x80001420 Failover ost1 to oleg349-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE FULL state after 0 sec at_max=600 PASS 31 (25s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 09:32:14 (1747402334) fail_loc=0x0000131d fail_val=10 fail_loc=0x726 Stopping /mnt/lustre-mds2 (opts:) on oleg349-server Stopping /mnt/lustre-mds1 (opts:) on oleg349-server fail_loc=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0001 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1285312 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2124 1285564 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1412 3605608 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1400 3604860 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2812 7210468 1% /mnt/lustre PASS 32 (17s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 09:32:31 (1747402351) Stopping /mnt/lustre-mds1 (opts:) on oleg349-server last_rcvd in /dev/mapper/mds1_flakey: incompat = 0x61c Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount (REPLAY_WAIT|REPLAY_LOCKS) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in REPLAY_LOCKS state after 0 sec oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping /mnt/lustre-mds1 (opts:) on oleg349-server last_rcvd in /dev/mapper/mds1_flakey: incompat = 0x61c Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre2 mount.lustre: mount oleg349-server@tcp:/lustre at /mnt/lustre2 failed: Input/output error Is the MGS running? oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec PASS 33 (32s) == replay-dual test complete, duration 2845 sec ========== 09:33:03 (1747402383) Stopping clients: oleg349-client.virtnet /mnt/lustre2 (opts:)