-----============= acceptance-small: replay-single ============----- Tue Jun 23 10:34:48 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg104-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-single.*ex': No such file or directory excepting tests: 110f 131b 59 36 === replay-single: start setup 10:35:17 (1782225317) === oleg104-client.virtnet: executing check_config_client /mnt/lustre oleg104-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg104-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cd107b68800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cd107b68800.idle_timeout=debug disable quota as required oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-single: finish setup 10:36:05 (1782225365) === Cleaning up AT ... == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 10:36:10 (1782225370) fail_loc=0x1701 Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 10:36:20 (1782225380) shut down facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 10:36:48 (1782225408) targets are mounted 10:36:48 (1782225408) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000400:0x2:0x0] 1 [0x240000401:0x2:0x0] total: 20 open/close in 0.58 seconds: 34.27 ops/second PASS 100a (62s) == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 10:37:12 (1782225432) fail_loc=0x119 Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server 10:37:22 (1782225442) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 10:38:02 (1782225482) targets are mounted 10:38:02 (1782225482) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000400:0x5:0x0] 1 [0x240000401:0x5:0x0] total: 20 open/close in 0.33 seconds: 60.62 ops/second PASS 100b (77s) == replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 ========================================================== 10:38:29 (1782225509) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1269460 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1720 1269584 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Stopping /mnt/lustre-mds2 (opts:) on oleg104-server Failover mds2 to oleg104-server oleg104-server.virtnet Start mds2: mount -t lustre -o localrecov -o abort_recov_mdt /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 total: 20 open/close in 0.65 seconds: 30.69 ops/second Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 10:39:32 (1782225572) shut down facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 10:40:02 (1782225602) targets are mounted 10:40:03 (1782225603) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 1 [0x240000bd0:0x3:0x0] 0 [0x200000403:0x3:0x0] total: 20 open/close in 0.41 seconds: 48.83 ops/second PASS 100c (117s) == replay-single test 100d: DNE: cancel update logs upon recovery abort ========================================================== 10:40:26 (1782225626) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d100d.replay-single total: 100 mkdir in 1.45 seconds: 69.20 ops/second lustre-MDT0000-osd [catalog]: [0x200000400:0x1:0x0] [index]: 00002 [logid]: [0x200000bd0:0x1:0x0] lustre-MDT0001-osp-MDT0000 [catalog]: [0x240000401:0x1:0x0] [index]: 00003 [logid]: [0x240000bd1:0x1:0x0] [index]: 00004 [logid]: [0x240000bd1:0x2:0x0] Stopping /mnt/lustre-mds1 (opts:) on oleg104-server Failover mds1 to oleg104-server oleg104-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 find: '/mnt/lustre/d100d.replay-single': No such file or directory find: '/mnt/lustre/d100d.replay-single': No such file or directory PASS 100d (64s) == replay-single test 100e: DNE: create striped dir on MDT0 and MDT1, fail MDT0, MDT1 ========================================================== 10:41:30 (1782225690) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2252 1269052 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1269196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2252 1269052 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1269196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000013a0:0x2:0x0] 1 [0x240001b71:0x2:0x0] Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 10:42:06 (1782225726) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 10:42:57 (1782225777) targets are mounted 10:42:57 (1782225777) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000013a0:0x2:0x0] 1 [0x240001b71:0x2:0x0] PASS 100e (110s) == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 10:43:20 (1782225800) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1268916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2228 1269076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Stopping /mnt/lustre-mds1 (opts:) on oleg104-server Failover mds1 to oleg104-server oleg104-server.virtnet Start mds1: mount -t lustre -o localrecov -o abort_recovery /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 101 (182s) == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 10:46:22 (1782225982) creating 7 files ... fail_loc=0x159 launch 7 chmod in parallel (10:46:26) ... fail_loc=0 done (10:47:22) /mnt/lustre/d102a.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102a.replay-single/file-7 has perms 0600 OK PASS 102a (67s) == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 10:47:30 (1782226050) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (10:47:33) ... fail_loc=0 done (10:48:28) /mnt/lustre/d102b.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102b.replay-single/file-7 has perms 0600 OK PASS 102b (66s) == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 10:48:36 (1782226116) creating 7 files ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1268844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2204 1269100 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605280 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605084 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210364 1% /mnt/lustre fail_loc=0x15a launch 7 chmod in parallel (10:48:53) ... fail_loc=0 Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server 10:49:03 (1782226143) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 10:49:41 (1782226181) targets are mounted 10:49:41 (1782226181) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec done (10:49:54) /mnt/lustre/d102c.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102c.replay-single/file-7 has perms 0600 OK PASS 102c (85s) == replay-single test 102d: check replay & reconstruction with multiple mod RPCs in flight ========================================================== 10:50:02 (1782226202) creating 7 files ... fail_loc=0x15a launch 7 chmod in parallel (10:50:06) ... fail_loc=0 Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 10:50:15 (1782226215) shut down facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 10:50:45 (1782226245) targets are mounted 10:50:45 (1782226245) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec done (10:51:00) /mnt/lustre/d102d.replay-single/file-1 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-2 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-3 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-4 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-5 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-6 has perms 0600 OK /mnt/lustre/d102d.replay-single/file-7 has perms 0600 OK PASS 102d (68s) == replay-single test 103: Check otr_next_id overflow ==== 10:51:09 (1782226269) fail_loc=0x80000162 total: 30 open/close in 0.74 seconds: 40.39 ops/second Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server 10:51:20 (1782226280) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 10:51:57 (1782226317) targets are mounted 10:51:57 (1782226317) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 103 (69s) == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 10:52:19 (1782226339) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1268900 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1269196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605280 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605084 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210364 1% /mnt/lustre Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server 10:52:37 (1782226357) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 10:53:24 (1782226404) targets are mounted 10:53:24 (1782226404) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110a.replay-single/striped_dir has type dir OK PASS 110a (91s) == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 10:53:50 (1782226430) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1268896 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1269200 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server 10:54:10 (1782226450) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 10:54:48 (1782226488) targets are mounted 10:54:48 (1782226488) facet_failover done pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre /mnt/lustre/d110b.replay-single/striped_dir has type dir OK PASS 110b (135s) == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 10:56:05 (1782226565) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1268892 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1269200 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 10:56:22 (1782226582) shut down facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 10:56:55 (1782226615) targets are mounted 10:56:55 (1782226615) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110c.replay-single/striped_dir has type dir OK PASS 110c (70s) == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 10:57:16 (1782226636) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1268900 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1269196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 10:57:37 (1782226657) shut down facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 10:58:10 (1782226690) targets are mounted 10:58:10 (1782226690) facet_failover done pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre /mnt/lustre/d110d.replay-single/striped_dir has type dir OK PASS 110d (132s) == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 10:59:28 (1782226768) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1269200 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 11:00:00 (1782226800) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:00:52 (1782226852) targets are mounted 11:00:52 (1782226852) facet_failover done pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre /mnt/lustre/d110e.replay-single/striped_dir has type dir OK PASS 110e (163s) SKIP: replay-single test_110f skipping excluded test 110f == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 11:02:13 (1782226933) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2104 1269200 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 11:02:46 (1782226966) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:03:39 (1782227019) targets are mounted 11:03:39 (1782227019) facet_failover done pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre /mnt/lustre/d110g.replay-single/striped_dir has type dir OK PASS 110g (161s) == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 11:04:54 (1782227094) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2456 1268848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1269188 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server 11:05:13 (1782227113) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 11:05:47 (1782227147) targets are mounted 11:05:47 (1782227147) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111a.replay-single/striped_dir: No such file or directory PASS 111a (75s) == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 11:06:09 (1782227169) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2460 1268844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1269184 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 11:06:30 (1782227190) shut down facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 11:07:05 (1782227225) targets are mounted 11:07:05 (1782227225) facet_failover done pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111b.replay-single/striped_dir: No such file or directory PASS 111b (134s) == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 11:08:23 (1782227303) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2504 1268800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1269184 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 11:09:02 (1782227342) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Failover mds2 to oleg104-server mount facets: mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 11:09:50 (1782227390) targets are mounted 11:09:50 (1782227390) facet_failover done pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111c.replay-single/striped_dir: No such file or directory PASS 111c (167s) == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 11:11:11 (1782227471) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2464 1268840 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2124 1269180 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre df:/mnt/lustre Not a Lustre filesystem Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 11:11:48 (1782227508) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Failover mds2 to oleg104-server mount facets: mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 11:12:40 (1782227560) targets are mounted 11:12:40 (1782227560) facet_failover done pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid pdsh@oleg104-client: oleg104-client: ssh exited with exit code 95 Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre Can't lstat /mnt/lustre/d111d.replay-single/striped_dir: No such file or directory PASS 111d (161s) == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 11:13:52 (1782227632) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2508 1268796 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2160 1269144 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2500 1268804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2152 1269152 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 11:14:31 (1782227671) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all oleg104-server: oleg104-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 11:15:22 (1782227722) targets are mounted 11:15:22 (1782227722) facet_failover done oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Can't lstat /mnt/lustre/d111e.replay-single/striped_dir: No such file or directory PASS 111e (113s) == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 11:15:45 (1782227745) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2468 1268836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2128 1269176 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2468 1268836 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2120 1269184 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg104-server Stopping /mnt/lustre-mds1 (opts:) on oleg104-server Failing mds2 on oleg104-server Stopping /mnt/lustre-mds2 (opts:) on oleg104-server 11:16:18 (1782227778) shut down facet: mds1 facet_host: oleg104-server facet_failover_host: oleg104-server facet: mds2 facet_host: oleg104-server facet_failover_host: oleg104-server Failover mds1 to oleg104-server mount facets: mds1 Failover mds2 to oleg104-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1