-----============= acceptance-small: replay-ost-single ============----- Tue Feb 27 12:09:47 EST 2024 excepting tests: === replay-ost-single: start setup 12:09:49 (1709053789) === oleg438-client.virtnet: executing check_config_client /mnt/lustre oleg438-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg438-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5c0b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5c0b000.idle_timeout=debug disable quota as required oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === replay-ost-single: finish setup 12:09:56 (1709053796) === /mnt/lustre/d0.replay-ost-single stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: 0 osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=1 osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=1 osp.lustre-OST0001-osc-MDT0001.prealloc_force_new_seq=1 osp.lustre-OST0000-osc-MDT0001.prealloc_force_new_seq=1 Creating to objid 33 on ost lustre-OST0001... Creating to objid 33 on ost lustre-OST0000... Creating to objid 33 on ost lustre-OST0001... Creating to objid 33 on ost lustre-OST0000... total: 33 open/close in 0.12 seconds: 273.63 ops/second total: 33 open/close in 0.12 seconds: 272.56 ops/second total: 33 open/close in 0.13 seconds: 260.12 ops/second total: 33 open/close in 0.13 seconds: 254.52 ops/second osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=0 osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=0 osp.lustre-OST0000-osc-MDT0001.prealloc_force_new_seq=0 osp.lustre-OST0001-osc-MDT0001.prealloc_force_new_seq=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 0a: target handle mismatch (bug 5317) ========================================================== 12:10:09 (1709053809) Stopping client oleg438-client.virtnet /mnt/lustre (opts:-f) fail_loc=0x80000211 Starting client: oleg438-client.virtnet: -o user_xattr,flock oleg438-server@tcp:/lustre /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] filesystem_summary: 3833116 1796 3605224 1% /mnt/lustre PASS 0a (13s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 0b: empty replay =============== 12:10:22 (1709053822) Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:10:23 (1709053823) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:10:36 (1709053836) targets are mounted 12:10:36 (1709053836) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 0b (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 1: touch ======================= 12:10:42 (1709053842) Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:10:44 (1709053844) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:10:57 (1709053857) targets are mounted 12:10:57 (1709053857) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /mnt/lustre/d0.replay-ost-single/f1.replay-ost-single has type file OK PASS 1 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 2: |x| 10 open(O_CREAT)s ======= 12:11:02 (1709053862) Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:11:04 (1709053864) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:11:17 (1709053877) targets are mounted 12:11:17 (1709053877) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 2 (19s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 3: Fail OST during write, with verification ========================================================== 12:11:22 (1709053882) Failing ost1 on oleg438-server 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 0.135635 s, 38.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:11:24 (1709053884) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:11:37 (1709053897) targets are mounted 12:11:37 (1709053897) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 3 (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 4: Fail OST during read, with verification ========================================================== 12:11:45 (1709053905) 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 0.243491 s, 21.5 MB/s Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:11:47 (1709053907) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:12:02 (1709053922) targets are mounted 12:12:02 (1709053922) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 4 (22s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 5: Fail OST during iozone ====== 12:12:09 (1709053929) iozone bg pid=18188 + iozone -i 0 -i 1 -+d -r 4 -s 1048576 -f /mnt/lustre/d0.replay-ost-single/f5.replay-ost-single tmppipe=/tmp/replay-ost-single.test_5.pipe iozone pid=18191 Iozone: Performance Test of File I/O Version $Revision: 3.483 $ Compiled for 64 bit mode. Build: linux-AMD64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, Vangel Bojaxhi, Ben England, Vikentsi Lapa, Alexey Skidanov, Sudhir Kumar. Run began: Tue Feb 27 12:12:10 2024 >>> I/O Diagnostic mode enabled. <<< Performance measurements are invalid in this mode. Record Size 4 kB File size set to 1048576 kB Command line used: iozone -i 0 -i 1 -+d -r 4 -s 1048576 -f /mnt/lustre/d0.replay-ost-single/f5.replay-ost-single Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:12:19 (1709053939) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:12:34 (1709053954) targets are mounted 12:12:34 (1709053954) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 1048576 4 20931 29471 369248 352582 iozone test complete. iozone rc=0 PASS 5 (104s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 6: Fail OST before obd_destroy ========================================================== 12:13:54 (1709054034) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. Waiting for MDT destroys to complete 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 0.212003 s, 24.7 MB/s /mnt/lustre/d0.replay-ost-single/f6.replay-ost-single lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 19 0x13 0x280000403 fail_loc=0x80000119 Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. before_free: 7662624 after_dd_free: 7657504 took 0 seconds Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:14:26 (1709054066) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:14:39 (1709054079) targets are mounted 12:14:39 (1709054079) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec affected facets: ost1 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg438-server: *.lustre-OST0000.recovery_status status: COMPLETE Can't lstat /mnt/lustre/d0.replay-ost-single/f6.replay-ost-single: No such file or directory Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. Waiting for MDT destroys to complete free_before: 7662624 free_after: 7662624 PASS 6 (54s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 7: Fail OST before obd_destroy ========================================================== 12:14:49 (1709054089) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. Waiting for MDT destroys to complete 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 0.430489 s, 12.2 MB/s before: 7662624 after_dd: 7657504 took 2 seconds UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 6932 3600088 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 8728 7205312 1% /mnt/lustre Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:15:09 (1709054109) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:15:26 (1709054126) targets are mounted 12:15:27 (1709054127) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec affected facets: ost1 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg438-server: *.lustre-OST0000.recovery_status status: COMPLETE Can't lstat /mnt/lustre/d0.replay-ost-single/f7.replay-ost-single: No such file or directory Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. Waiting for MDT destroys to complete before: 7662624 after: 7662624 PASS 7 (49s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 8a: Verify redo io: redo io when get -EINPROGRESS error ========================================================== 12:15:40 (1709054140) 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 0.0540509 s, 97.0 MB/s fail_loc=0x230 fail_loc=0 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 33.886 s, 155 kB/s PASS 8a (37s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 8b: Verify redo io: redo io should success after recovery ========================================================== 12:16:20 (1709054180) 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 0.0475459 s, 110 MB/s fail_loc=0x230 Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:16:42 (1709054202) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:16:57 (1709054217) targets are mounted 12:16:57 (1709054217) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 54.3824 s, 96.4 kB/s PASS 8b (59s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 8c: Verify redo io: redo io should fail after eviction ========================================================== 12:17:23 (1709054243) 1280+0 records in 1280+0 records out 5242880 bytes (5.2 MB) copied, 0.0682575 s, 76.8 MB/s fail_loc=0x230 dd: error writing '/mnt/lustre/d0.replay-ost-single/f8c.replay-ost-single': Input/output error 1+0 records in 0+0 records out 0 bytes (0 B) copied, 21.3971 s, 0.0 kB/s fail_loc=0 /tmp/verify-7328 /mnt/lustre/d0.replay-ost-single/f8c.replay-ost-single differ: byte 1, line 1 PASS 8c (47s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 8d: Verify redo creation on -EINPROGRESS ========================================================== 12:18:14 (1709054294) fail_loc=0x187 fail_loc=0 File: '/mnt/lustre/d0.replay-ost-single/f8d.replay-ost-single' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056727 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-02-27 12:18:15.000000000 -0500 Modify: 2024-02-27 12:18:15.000000000 -0500 Change: 2024-02-27 12:18:15.000000000 -0500 Birth: - fail_loc=0x187 fail_loc=0 Succeed in opening file "/mnt/lustre/d0.replay-ost-single/f8d.replay-ost-single"(flags=O_RDWR) File: '/mnt/lustre/d0.replay-ost-single/f8d.replay-ost-single' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056728 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-02-27 12:18:37.000000000 -0500 Modify: 2024-02-27 12:18:37.000000000 -0500 Change: 2024-02-27 12:18:37.000000000 -0500 Birth: - PASS 8d (46s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 8e: Verify that ptlrpc resends request on -EINPROGRESS ========================================================== 12:19:02 (1709054342) fail_loc=0x231 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre PASS 8e (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 9: Verify that no req deadline happened during recovery ========================================================== 12:19:29 (1709054369) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0294616 s, 35.6 MB/s UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3604160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7209384 1% /mnt/lustre 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0160674 s, 65.3 MB/s fail_loc=0x00000714 fail_val=20 Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:19:36 (1709054376) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:19:55 (1709054395) targets are mounted 12:19:55 (1709054395) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (70s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 10: conflicting PW & PR locks on a client ========================================================== 12:20:43 (1709054443) 10+0 records in 10+0 records out 5120 bytes (5.1 kB) copied, 0.0132738 s, 386 kB/s fail_val=60 fail_loc=0x414 Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:20:48 (1709054448) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:21:04 (1709054464) targets are mounted 12:21:04 (1709054464) facet_failover done fail_loc=0x32a File: '/mnt/lustre/d0.replay-ost-single/f10.replay-ost-single' Size: 5120 Blocks: 0 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205306056733 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-02-27 12:20:44.000000000 -0500 Modify: 2024-02-27 12:20:44.000000000 -0500 Change: 2024-02-27 12:20:44.000000000 -0500 Birth: - PASS 10 (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 12a: glimpse after OST failover to a missing object ========================================================== 12:21:50 (1709054510) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1844 1285844 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2844 3604176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 4640 7209400 1% /mnt/lustre total: 500 open/close in 3.65 seconds: 137.03 ops/second total: 500 open/close in 3.36 seconds: 148.89 ops/second total: 500 open/close in 3.74 seconds: 133.59 ops/second total: 500 open/close in 4.14 seconds: 120.81 ops/second total: 500 open/close in 3.95 seconds: 126.44 ops/second total: 500 open/close in 4.20 seconds: 119.05 ops/second total: 500 open/close in 4.13 seconds: 121.02 ops/second total: 500 open/close in 4.28 seconds: 116.91 ops/second total: 500 open/close in 4.13 seconds: 121.01 ops/second total: 500 open/close in 4.63 seconds: 108.07 ops/second Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:22:55 (1709054575) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:23:11 (1709054591) targets are mounted 12:23:11 (1709054591) facet_failover done starting wait for ls -l PASS 12a (105s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-ost-single test 12b: write after OST failover to a missing object ========================================================== 12:23:37 (1709054617) striped dir -i0 -c2 -H crush /mnt/lustre/d12b.replay-ost-single UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2228 1285460 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1285604 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 2844 3604176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 4640 7209400 1% /mnt/lustre total: 500 open/close in 2.05 seconds: 243.33 ops/second total: 500 open/close in 2.14 seconds: 233.20 ops/second total: 500 open/close in 2.84 seconds: 176.16 ops/second total: 500 open/close in 2.65 seconds: 188.47 ops/second total: 500 open/close in 3.64 seconds: 137.48 ops/second total: 500 open/close in 4.32 seconds: 115.65 ops/second total: 500 open/close in 4.51 seconds: 110.77 ops/second total: 500 open/close in 4.06 seconds: 123.03 ops/second total: 500 open/close in 3.24 seconds: 154.37 ops/second total: 500 open/close in 3.22 seconds: 155.42 ops/second fail_loc=0x16e fail_val=10 Failing ost1 on oleg438-server Stopping /mnt/lustre-ost1 (opts:) on oleg438-server 12:24:28 (1709054668) shut down Failover ost1 to oleg438-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 12:24:45 (1709054685) targets are mounted 12:24:45 (1709054685) facet_failover done 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00817633 s, 501 kB/s PASS 12b (99s) debug_raw_pointers=0 debug_raw_pointers=0 == replay-ost-single test complete, duration 930 sec ===== 12:25:17 (1709054717) === replay-ost-single: start cleanup 12:25:18 (1709054718) === === replay-ost-single: finish cleanup 12:25:18 (1709054718) ===