-----============= acceptance-small: sanity-flr ============----- Sat May 9 06:14:22 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg226-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:14:45 (1778321685) === oleg226-client.virtnet: executing check_config_client /mnt/lustre oleg226-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg226-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8989c8d93800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8989c8d93800.idle_timeout=debug disable quota as required oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 06:15:17 (1778321717) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7578] == sanity-flr test 0a: lfs mirror create with -N option == 06:15:19 (1778321719) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:15:34 (1778321734) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:15:37 (1778321737) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:15:39 (1778321739) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:16:01 (1778321761) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:16:04 (1778321764) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 06:16:08 (1778321768) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8989c8d93800.stats=clear osc.lustre-OST0001-osc-ffff8989c8d93800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:16:15 (1778321775) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:16:26 (1778321786) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:16:34 (1778321794) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:16:37 (1778321797) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:16:44 (1778321804) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 06:16:53 (1778321813) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:17:00 (1778321820) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:17:03 (1778321823) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:17:10 (1778321830) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:17:21 (1778321841) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:17:27 (1778321847) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.77671 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170746 s, 6.1 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:17:39 (1778321859) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.464913 s, 45.1 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:17:46 (1778321866) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:17:56 (1778321876) Stopping /mnt/lustre-ost1 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8989c8d93800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8989c8d93800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg226-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8989c8d93800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8989c8d93800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg226-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (65s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:19:01 (1778321941) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.203482 s, 20.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8989c8d93800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8989c8d93800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg226-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (33s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:19:34 (1778321974) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8989c8d93800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8989c8d93800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg226-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8989c8d93800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8989c8d93800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg226-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:20:56 (1778322056) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.86427 s, 8.9 MB/s -rw-r--r-- 1 root root 52428800 May 9 06:21 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.362s user 0m0.000s sys 0m0.167s Stopping /mnt/lustre-ost1 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8989c8d93800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8989c8d93800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.073s user 0m0.001s sys 0m0.175s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg226-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8989c8d93800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8989c8d93800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.928s user 0m0.001s sys 0m0.158s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg226-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg226-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8989c8d93800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8989c8d93800.ost_server_uuid in FULL state after 0 sec PASS 33b (86s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:22:22 (1778322142) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:22:26 (1778322146) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:22:29 (1778322149) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:22:32 (1778322152) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 06:22:39 (1778322159) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.553481 s, 22.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:22:52 (1778322172) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397097 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405259 s, 25.9 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:22:59 (1778322179) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:23:07 (1778322187) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00121495 s, 843 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011002 s, 931 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000935403 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00149466 s, 685 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 06:23:18 (1778322198) create test files with size 15756645 .. 784ef254fdf7b18e41e506cb269f0208 - 863a92c37838960be2e3b4caa002fd69 - 4218ccf7f877a91e2b12702a4dc1004c - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy ..