-----============= acceptance-small: sanity-flr ============----- Sun Nov 16 22:33:36 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg122-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:33:54 (1763350434) === oleg122-client.virtnet: executing check_config_client /mnt/lustre oleg122-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg122-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff936ed8f06800.idle_timeout=debug osc.lustre-OST0001-osc-ffff936ed8f06800.idle_timeout=debug disable quota as required oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:34:25 (1763350465) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7593] == sanity-flr test 0a: lfs mirror create with -N option == 22:34:27 (1763350467) PASS 0a (13s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:34:40 (1763350480) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:34:43 (1763350483) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:34:46 (1763350486) PASS 0d (20s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:35:06 (1763350506) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:35:09 (1763350509) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 22:35:12 (1763350512) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff936ed8f06800.stats=clear osc.lustre-OST0001-osc-ffff936ed8f06800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:35:19 (1763350519) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:35:28 (1763350528) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:35:34 (1763350534) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:35:37 (1763350537) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (7s) == sanity-flr test 2: create components from existing files ========================================================== 22:35:44 (1763350544) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:35:49 (1763350549) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:35:52 (1763350552) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:35:58 (1763350558) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:36:07 (1763350567) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:36:12 (1763350572) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.261749 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147325 s, 7.1 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:36:20 (1763350580) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.355534 s, 44.2 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:36:27 (1763350587) Stopping /mnt/lustre-ost1 (opts:) on oleg122-server oleg122-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff936ed8f06800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936ed8f06800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg122-server: oleg122-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg122-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg122-server oleg122-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff936ed8f06800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff936ed8f06800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg122-server: oleg122-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg122-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (65s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:37:32 (1763350652) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.429883 s, 17.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg122-server oleg122-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff936ed8f06800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936ed8f06800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg122-server: oleg122-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg122-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (37s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:38:09 (1763350689) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg122-server oleg122-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff936ed8f06800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936ed8f06800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg122-server: oleg122-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg122-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg122-server oleg122-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff936ed8f06800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff936ed8f06800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg122-server: oleg122-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg122-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (62s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:39:11 (1763350751) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.27882 s, 12.3 MB/s -rw-r--r-- 1 root root 52428800 Nov 16 22:39 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.096s user 0m0.001s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg122-server oleg122-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff936ed8f06800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936ed8f06800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.026s user 0m0.001s sys 0m0.182s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg122-server: oleg122-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg122-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg122-server oleg122-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff936ed8f06800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff936ed8f06800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.124s user 0m0.001s sys 0m0.191s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg122-server: oleg122-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg122-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg122-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff936ed8f06800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff936ed8f06800.ost_server_uuid in FULL state after 0 sec PASS 33b (78s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:40:29 (1763350829) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:40:33 (1763350833) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:40:36 (1763350836) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:40:40 (1763350840) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 22:40:47 (1763350847) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.6361 s, 19.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:41:00 (1763350860) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369616 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338677 s, 31.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:41:07 (1763350867) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:41:14 (1763350874) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00139286 s, 735 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00469835 s, 218 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000891668 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0019325 s, 530 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 22:41:26 (1763350886) create test files with size 15751401 .. 3611a948e223147425701e7438889527 - e0886872287aec5d3e042ea340dfe895 - 1eaee95163765233832f3edd638fa7f8 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg122-client.virtnet /mnt/lustre (opts:) Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 22:41:52 (1763350912) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:42:06 (1763350926) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:42:13 (1763350933) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145758 s, 7.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306352 s, 34.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 22:42:22 (1763350942) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147873 s, 14.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.395464 s, 10.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.232837 s, 9.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 22:42:37 (1763350957) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:42:40 (1763350960) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:42:44 (1763350964) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407713 s, 25.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 22:42:51 (1763350971) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:42:54 (1763350974) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:43:03 (1763350983) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.208721 s, 50.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg122-client.virtnet /mnt/lustre (opts:) Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:43:15 (1763350995) llite.lustre-ffff936ec5272800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.289048 s, 36.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg122-client.virtnet /mnt/lustre (opts:) Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff936ec5641800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:43:28 (1763351008) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:44:04 (1763351044) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:44:34 (1763351074) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:44:37 (1763351077) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.20046 s, 15.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0910206 s, 34.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x854a171a (stale) Mirror 2: 0x854a171a (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 28d9edc267a5ea3c03322519c31113f1 - mirror 1 checksum: e6fc5e58a1b1ff6307037851307f1e8f - mirror 2 checksum: 28d9edc267a5ea3c03322519c31113f1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 22:44:39 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 22:44:39 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 28d9edc267a5ea3c03322519c31113f1 - mirror 1 checksum: e6fc5e58a1b1ff6307037851307f1e8f - mirror 2 checksum: e6fc5e58a1b1ff6307037851307f1e8f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 22:44:39 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 22:44:39 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:44:48 (1763351088) Failing mds1 on oleg122-server Stopping /mnt/lustre-mds1 (opts:) on oleg122-server 22:44:54 (1763351094) shut down facet: mds1 facet_host: oleg122-server facet_failover_host: oleg122-server Failover mds1 to oleg122-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 22:45:25 (1763351125) targets are mounted 22:45:25 (1763351125) facet_failover done oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335684 s, 31.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (50s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:45:38 (1763351138) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:45:41 (1763351141) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:45:44 (1763351144) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0118279 s, 1.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:45:51 (1763351151) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:45:54 (1763351154) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:45:58 (1763351158) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:46:21 (1763351181) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:46:48 (1763351208) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:47:12 (1763351232) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:47:37 (1763351257) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 22:47:46 (1763351266) mirror create pid 52314 mirror split pid 52315 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52314 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52315 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 22:48:53 (1763351333) Seed set to 1335 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1650 000005 1763351335.428612 trunc from 00000000 to 0x00164f (0x1650 bytes) 000007 1763351335.470749 mapread 0x000c59 thru 0x00164f (0x9f7 bytes) 000008 1763351335.499917 read 0x001537 thru 0x00164f (0x119 bytes) truncating to largest ever: 0x3c602 000009 1763351335.501789 trunc from 0x001650 to 0x03c601 (0x3afb2 bytes) 000011 1763351335.538583 mapread 0x038ee8 thru 0x039401 (0x51a bytes) 000014 1763351335.575525 mapread 0x0157bd thru 0x017218 (0x1a5c bytes) 000015 1763351335.596760 mapread 0x00e6c7 thru 0x01a440 (0xbd7a bytes) 000016 1763351335.599914 mapwrite 0x02417f thru 0x0276e2 (0x3564 bytes) 000017 1763351335.665770 read 0x0357b8 thru 0x039e84 (0x46cd bytes) 000018 1763351335.666972 read 0x0225fa thru 0x025bdd (0x35e4 bytes) 000020 1763351335.669009 read 0x0330a2 thru 0x03c601 (0x9560 bytes) 000021 1763351335.670088 write 0x0074db thru 0x014da3 (0xd8c9 bytes) 000023 1763351335.683490 read 0x03423b thru 0x03c601 (0x83c7 bytes) 000027 1763351335.686934 write 0x01a75d thru 0x0227a7 (0x804b bytes) 000031 1763351335.695048 mapread 0x00d955 thru 0x00e626 (0xcd2 bytes) 000032 1763351335.698511 mapread 0x03c5f6 thru 0x03c601 (0xc bytes) 000035 1763351335.701186 read 0x018565 thru 0x020f4f (0x89eb bytes) 000037 1763351335.703524 write 0x00a48a thru 0x0143dc (0x9f53 bytes) 000040 1763351335.706136 mapread 0x028e6f thru 0x02a16a (0x12fc bytes) 000042 1763351335.710397 mapread 0x0048b6 thru 0x0134fd (0xec48 bytes) 000048 1763351335.715425 mapread 0x009632 thru 0x0129a8 (0x9377 bytes) 000052 1763351335.719422 read 0x01b2ef thru 0x020d4d (0x5a5f bytes) 000055 1763351335.721340 mapwrite 0x01ff5e thru 0x02d1bf (0xd262 bytes) 000057 1763351335.843125 trunc from 0x03c602 to 0x013ed9 (0x28728 bytes) 000058 1763351335.872081 mapwrite 0x031420 thru 0x03fab6 (0xe697 bytes) 000060 1763351335.994937 trunc from 0x03fab7 to 0x019461 (0x26655 bytes) 000061 1763351336.037079 read 0x018d2a thru 0x019461 (0x738 bytes) 000062 1763351336.037729 trunc from 0x019462 to 0x0251df (0xbd7e bytes) 000063 1763351336.076693 write 0x014b98 thru 0x020614 (0xba7d bytes) 000065 1763351336.095844 write 0x036e98 thru 0x03d2f3 (0x645c bytes) 000067 1763351336.097841 mapwrite 0x03b6fd thru 0x03e19e (0x2aa2 bytes) 000068 1763351336.294563 mapread 0x03b315 thru 0x03c729 (0x1415 bytes) 000070 1763351336.295788 mapread 0x025774 thru 0x028087 (0x2914 bytes) 000071 1763351336.296555 read 0x033814 thru 0x03e19e (0xa98b bytes) 000072 1763351336.297095 read 0x0332e9 thru 0x03e19e (0xaeb6 bytes) 000074 1763351336.299129 read 0x002fda thru 0x0041c4 (0x11eb bytes) 000076 1763351336.310575 mapwrite 0x0182be thru 0x01946c (0x11af bytes) 000077 1763351336.407289 mapwrite 0x006e63 thru 0x0100f2 (0x9290 bytes) 000079 1763351336.514683 mapwrite 0x03b576 thru 0x03ffff (0x4a8a bytes) 000080 1763351336.616338 mapread 0x01936c thru 0x027ea7 (0xeb3c bytes) 000083 1763351336.619122 read 0x029879 thru 0x037b02 (0xe28a bytes) 000086 1763351336.622630 write 0x0381db thru 0x038dd6 (0xbfc bytes) 000087 1763351336.625097 mapwrite 0x009484 thru 0x00aedf (0x1a5c bytes) 000088 1763351336.680260 mapwrite 0x036913 thru 0x03ffff (0x96ed bytes) 000089 1763351336.744940 mapread 0x013285 thru 0x02134e (0xe0ca bytes) 000092 1763351336.747156 mapread 0x008e09 thru 0x01708a (0xe282 bytes) 000095 1763351336.750042 read 0x00c175 thru 0x01bbf8 (0xfa84 bytes) 000096 1763351336.752017 mapwrite 0x01119c thru 0x018678 (0x74dd bytes) 000097 1763351336.796142 trunc from 0x040000 to 0x024af0 (0x1b50f bytes) 000098 1763351336.814550 mapread 0x01c082 thru 0x024af0 (0x8a6f bytes) 000099 1763351336.815642 write 0x01c196 thru 0x0218a9 (0x5714 bytes) 000103 1763351336.818147 trunc from 0x024af1 to 0x009cc9 (0x1ae27 bytes) 000109 1763351336.838665 read 0x0003ed thru 0x009cc9 (0x98dd bytes) 000112 1763351336.840984 read 0x005a18 thru 0x009cc9 (0x42b2 bytes) 000114 1763351336.843028 mapwrite 0x00f6c4 thru 0x016766 (0x70a3 bytes) 000115 1763351336.951553 trunc from 0x016767 to 0x02834e (0x11be8 bytes) 000117 1763351336.989055 read 0x0162ec thru 0x01a075 (0x3d8a bytes) 000119 1763351337.002186 mapwrite 0x03ed27 thru 0x03ffff (0x12d9 bytes) 000120 1763351337.076410 write 0x03f6ec thru 0x03ffff (0x914 bytes) 000121 1763351337.077397 mapread 0x001a8d thru 0x004fe7 (0x355b bytes) 000122 1763351337.078202 read 0x00c5e8 thru 0x01b5d0 (0xefe9 bytes) 000126 1763351337.085509 MIRROR_EXTEND to 2 mirrors 000127 1763351337.783119 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000128 1763351337.964708 mapread 0x02bc0b thru 0x02df8e (0x2384 bytes) 000130 1763351337.989180 read 0x01f676 thru 0x01fffa (0x985 bytes) 000131 1763351337.996949 mapread 0x027ef0 thru 0x030ba5 (0x8cb6 bytes) 000136 1763351338.005435 MIRROR_EXTEND to 2 mirrors 000137 1763351338.857191 MIRROR_RESYNC 2 mirrors 000138 1763351339.202010 mapwrite 0x02c374 thru 0x03a3f5 (0xe082 bytes) 000139 1763351339.414993 read 0x01d1d9 thru 0x0201fe (0x3026 bytes) 000140 1763351339.426824 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xab6cdcf2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa40fd3d6 Mirror 3: 0xa40fd3d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000141 1763351340.272824 read 0x01cce9 thru 0x022f76 (0x628e bytes) 000142 1763351340.285320 read 0x01fcc9 thru 0x02414f (0x4487 bytes) 000143 1763351340.304647 MIRROR_RESYNC 3 mirrors 000144 1763351340.997817 trunc from 0x040000 to 0x02197a (0x1e685 bytes) 000145 1763351341.163591 read 0x0045a0 thru 0x008fdf (0x4a40 bytes) 000146 1763351341.178743 mapwrite 0x006648 thru 0x015af8 (0xf4b1 bytes) 000147 1763351341.273835 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2197b): Mirror 2: 0x580f1d47 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2197b): Mirror 2: 0x580f1d47 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2197b: skipped 000149 1763351341.659525 mapwrite 0x01ad31 thru 0x0293b3 (0xe683 bytes) 000150 1763351341.772012 mapwrite 0x019c0e thru 0x026401 (0xc7f4 bytes) 000153 1763351341.843747 mapwrite 0x03dd1b thru 0x03ffff (0x22e5 bytes) 000154 1763351341.925339 trunc from 0x040000 to 0x03573c (0xa8c3 bytes) 000156 1763351341.947134 mapread 0x0260c5 thru 0x02c4b3 (0x63ef bytes) 000158 1763351341.957110 write 0x03329c thru 0x03ed6e (0xbad3 bytes) 000159 1763351341.964651 read 0x00efde thru 0x01e6a7 (0xf6ca bytes) 000161 1763351341.975945 read 0x02fd25 thru 0x030968 (0xc44 bytes) 000163 1763351341.976941 mapwrite 0x00e157 thru 0x01aac4 (0xc96e bytes) 000165 1763351342.053691 trunc from 0x03ed6f to 0x01f48d (0x1f8e1 bytes) 000167 1763351342.078684 read 0x011593 thru 0x01d300 (0xbd6e bytes) 000168 1763351342.084609 MIRROR_RESYNC 2 mirrors 000169 1763351342.601111 write 0x013984 thru 0x016f90 (0x360d bytes) 000171 1763351342.698274 write 0x030b6d thru 0x03f3d4 (0xe868 bytes) 000173 1763351342.700711 mapwrite 0x01098a thru 0x015102 (0x4779 bytes) 000175 1763351342.764564 write 0x03311c thru 0x03e03c (0xaf21 bytes) 000176 1763351342.765642 trunc from 0x03f3d5 to 0x02a9b8 (0x14a1c bytes) 000177 1763351342.839209 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2a9b9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a9b9): Mirror 2: 0x7babcef8 (stale) CRC-32 checksum value for chunk [0, 0x2a9b9): Mirror 1: 0x332f8ae6 Mirror 3: 0x332f8ae6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a9b9: skipped 000178 1763351343.655609 trunc from 0x02a9b9 to 0x00269f (0x28319 bytes) 000179 1763351343.761110 read 0x001fb9 thru 0x00269f (0x6e7 bytes) 000180 1763351343.770977 read 0x0001e4 thru 0x00269f (0x24bc bytes) 000183 1763351343.792611 MIRROR_RESYNC 3 mirrors 000185 1763351344.615136 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000186 1763351344.981731 mapread 0x002199 thru 0x00269f (0x507 bytes) 000190 1763351344.993480 mapread 0x0023fd thru 0x00269f (0x2a3 bytes) 000192 1763351344.995634 mapwrite 0x00b655 thru 0x01b1f2 (0xfb9e bytes) 000195 1763351345.164776 write 0x02bd84 thru 0x032cf9 (0x6f76 bytes) 000197 1763351345.167449 mapread 0x002742 thru 0x007a37 (0x52f6 bytes) 000199 1763351345.177426 trunc from 0x032cfa to 0x02d46c (0x588d bytes) 000200 1763351345.206445 mapread 0x0158e5 thru 0x01a352 (0x4a6e bytes) 000202 1763351345.208564 read 0x001994 thru 0x005b2c (0x4199 bytes) 000205 1763351345.210696 mapread 0x000719 thru 0x00f3e5 (0xeccd bytes) 000206 1763351345.235918 read 0x00da75 thru 0x01b476 (0xda02 bytes) 000207 1763351345.243475 MIRROR_RESYNC 2 mirrors 000208 1763351345.927100 read 0x027aa4 thru 0x02c3f5 (0x4952 bytes) 000211 1763351345.940402 write 0x031c1d thru 0x0349cf (0x2db3 bytes) 000213 1763351346.040242 MIRROR_RESYNC 2 mirrors 000214 1763351346.651645 write 0x023a3c thru 0x032418 (0xe9dd bytes) 000215 1763351346.726021 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000216 1763351346.909644 mapread 0x0343dd thru 0x0349cf (0x5f3 bytes) 000218 1763351346.919843 read 0x034703 thru 0x0349cf (0x2cd bytes) 000220 1763351346.920840 mapread 0x022b18 thru 0x02a550 (0x7a39 bytes) 000221 1763351346.954156 read 0x00afc6 thru 0x0165f8 (0xb633 bytes) 000222 1763351346.974131 trunc from 0x0349d0 to 0x02112a (0x138a5 bytes) 000225 1763351346.997371 trunc from 0x02112b to 0x026d7a (0x5c50 bytes) 000226 1763351347.021779 trunc from 0x026d7b to 0x035d48 (0xefce bytes) 000227 1763351347.039859 read 0x02ddfe thru 0x035d48 (0x7f4b bytes) 000228 1763351347.053560 trunc from 0x035d49 to 0x03159b (0x47ad bytes) 000232 1763351347.075008 write 0x03bfa9 thru 0x03ffff (0x4057 bytes) 000233 1763351347.076089 mapread 0x0243aa thru 0x026506 (0x215d bytes) 000234 1763351347.077536 read 0x011704 thru 0x0182da (0x6bd7 bytes) 000237 1763351347.080194 mapread 0x01a310 thru 0x0294a8 (0xf199 bytes) 000239 1763351347.081718 mapread 0x00e511 thru 0x01c247 (0xdd37 bytes) 000241 1763351347.087684 mapread 0x020da4 thru 0x022129 (0x1386 bytes) 000242 1763351347.089600 mapread 0x00265f thru 0x01201f (0xf9c1 bytes) 000249 1763351347.100348 MIRROR_EXTEND to 2 mirrors 000250 1763351347.789108 trunc from 0x040000 to 0x003b4c (0x3c4b3 bytes) 000251 1763351347.956959 trunc from 0x003b4d to 0x035b93 (0x32047 bytes) 000253 1763351347.998000 write 0x01e13f thru 0x02ab3d (0xc9ff bytes) 000254 1763351348.011540 mapread 0x0206b4 thru 0x029d23 (0x9670 bytes) 000256 1763351348.014828 write 0x029e43 thru 0x0336fe (0x98bc bytes) 000260 1763351348.027215 trunc from 0x035b94 to 0x0324ec (0x36a7 bytes) 000262 1763351348.047479 trunc from 0x0324ed to 0x00390b (0x2ebe1 bytes) 000265 1763351348.074771 mapwrite 0x00ccb4 thru 0x0126bb (0x5a08 bytes) 000266 1763351348.170930 write 0x001e93 thru 0x0030ac (0x121a bytes) 000267 1763351348.182831 trunc from 0x0126bc to 0x01a776 (0x80bb bytes) 000268 1763351348.207466 read 0x00ade0 thru 0x00fd35 (0x4f56 bytes) 000269 1763351348.208262 mapwrite 0x00301f thru 0x00865b (0x563d bytes) 000273 1763351348.275948 read 0x010923 thru 0x015381 (0x4a5f bytes) 000275 1763351348.293105 write 0x02f490 thru 0x033a16 (0x4587 bytes) 000276 1763351348.303136 read 0x0206ec thru 0x027402 (0x6d17 bytes) 000278 1763351348.319456 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33a17): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a17): Mirror 2: 0xbc0517f4 (stale) CRC-32 checksum value for chunk [0, 0x33a17): Mirror 1: 0x2f90e9f4 Mirror 3: 0x2f90e9f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a17: skipped 000280 1763351349.232036 MIRROR_RESYNC 3 mirrors 000282 1763351349.956633 MIRROR_EXTEND to 4 mirrors 000283 1763351350.700733 write 0x03aa60 thru 0x03ffff (0x55a0 bytes) 000284 1763351350.819924 trunc from 0x040000 to 0x034e67 (0xb198 bytes) 000285 1763351350.911218 trunc from 0x034e68 to 0x00a4d1 (0x2a996 bytes) 000287 1763351350.933569 read 0x00671f thru 0x00a4d1 (0x3db3 bytes) 000288 1763351350.944630 read 0x008237 thru 0x008e51 (0xc1b bytes) 000289 1763351350.945235 read 0x000e3b thru 0x001378 (0x53e bytes) 000290 1763351350.959194 write 0x02f727 thru 0x0330eb (0x39c5 bytes) 000296 1763351350.963541 read 0x026054 thru 0x02f0f4 (0x90a1 bytes) 000297 1763351350.979307 read 0x02e52f thru 0x0330eb (0x4bbd bytes) 000298 1763351350.980129 write 0x035234 thru 0x0394c3 (0x4290 bytes) 000302 1763351350.984629 trunc from 0x0394c4 to 0x02a603 (0xeec0 bytes) 000303 1763351351.011630 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a604): Mirror 2: 0x70776970 (stale) Mirror 3: 0x70776970 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a604): Mirror 2: 0x70776970 (stale) Mirror 3: 0x70776970 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a604: skipped 000305 1763351351.433119 mapwrite 0x01dcc4 thru 0x021497 (0x37d4 bytes) 000306 1763351351.521115 trunc from 0x02a604 to 0x02205f (0x85a4 bytes) 000307 1763351351.561427 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22060): Mirror 3: 0x5149ea3d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22060): Mirror 3: 0x5149ea3d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22060: skipped 000309 1763351351.873342 read 0x00a711 thru 0x00b9e1 (0x12d1 bytes) 000310 1763351351.880169 trunc from 0x022060 to 0x02df58 (0xbef9 bytes) 000313 1763351351.899485 trunc from 0x02df59 to 0x02c4ef (0x1a69 bytes) 000315 1763351351.917292 trunc from 0x02c4f0 to 0x005cd6 (0x26819 bytes) 000318 1763351351.936342 trunc from 0x005cd7 to 0x0120b1 (0xc3db bytes) 000319 1763351351.953796 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000322 1763351352.135674 trunc from 0x0120b2 to 0x03050d (0x1e45c bytes) 000328 1763351352.166850 trunc from 0x03050e to 0x00201d (0x2e4f0 bytes) 000330 1763351352.190772 trunc from 0x00201e to 0x01bfa3 (0x19f86 bytes) 000331 1763351352.212717 mapread 0x0028cb thru 0x00926e (0x69a4 bytes) 000332 1763351352.234980 write 0x03714f thru 0x03ffff (0x8eb1 bytes) 000333 1763351352.236829 write 0x0176df thru 0x017a24 (0x346 bytes) 000340 1763351352.249836 mapread 0x0177b2 thru 0x01b591 (0x3de0 bytes) 000341 1763351352.254021 write 0x027c81 thru 0x029cb2 (0x2032 bytes) 000342 1763351352.263433 mapread 0x01046b thru 0x01f710 (0xf2a6 bytes) 000345 1763351352.266421 write 0x00bc7b thru 0x00ff8a (0x4310 bytes) 000346 1763351352.268995 write 0x017123 thru 0x022749 (0xb627 bytes) 000350 1763351352.272103 trunc from 0x040000 to 0x035e3e (0xa1c1 bytes) 000353 1763351352.296411 read 0x0304a4 thru 0x035e3e (0x599b bytes) 000354 1763351352.307848 mapread 0x020af8 thru 0x0262d8 (0x57e1 bytes) 000356 1763351352.310274 mapread 0x01d6d9 thru 0x02c939 (0xf261 bytes) 000359 1763351352.312821 mapwrite 0x027535 thru 0x02a582 (0x304e bytes) 000360 1763351352.367854 trunc from 0x035e3f to 0x0165ef (0x1f84f bytes) 000363 1763351352.383794 trunc from 0x0165f0 to 0x0352cb (0x1ecdc bytes) 000364 1763351352.417254 trunc from 0x0352cc to 0x031ca3 (0x3628 bytes) 000365 1763351352.441467 trunc from 0x031ca4 to 0x01f00a (0x12c99 bytes) 000366 1763351352.481992 trunc from 0x01f00b to 0x001a8a (0x1d580 bytes) 000369 1763351352.508837 read 0x001492 thru 0x001a8a (0x5f9 bytes) 000370 1763351352.509524 read 0x000f26 thru 0x001a8a (0xb65 bytes) 000371 1763351352.515621 mapwrite 0x000fae thru 0x007ae8 (0x6b3b bytes) 000372 1763351352.597391 read 0x003a3e thru 0x007ae8 (0x40ab bytes) 000373 1763351352.598093 trunc from 0x007ae9 to 0x008b6f (0x1087 bytes) 000374 1763351352.631946 mapwrite 0x018287 thru 0x023c0d (0xb987 bytes) 000379 1763351352.758274 write 0x03279a thru 0x03ffff (0xd866 bytes) 000380 1763351352.767765 mapwrite 0x002fc8 thru 0x005ea9 (0x2ee2 bytes) 000383 1763351352.842698 mapwrite 0x03c97e thru 0x03ffff (0x3682 bytes) truncating to largest ever: 0x3d603 000386 1763351352.915653 trunc from 0x040000 to 0x03d602 (0x29fd bytes) 000387 1763351352.937683 write 0x002f35 thru 0x00440f (0x14db bytes) 000389 1763351352.939552 mapread 0x02620b thru 0x02d54f (0x7345 bytes) 000390 1763351352.954251 mapwrite 0x027afc thru 0x034006 (0xc50b bytes) 000391 1763351353.004351 read 0x00ba32 thru 0x0147f6 (0x8dc5 bytes) 000392 1763351353.005045 mapwrite 0x023461 thru 0x02ddc5 (0xa965 bytes) 000393 1763351353.052278 mapwrite 0x00f9a4 thru 0x015bbe (0x621b bytes) 000395 1763351353.099289 write 0x00bceb thru 0x017cd5 (0xbfeb bytes) 000396 1763351353.100602 mapwrite 0x01a551 thru 0x020e48 (0x68f8 bytes) 000399 1763351353.149682 trunc from 0x03d603 to 0x018739 (0x24ec9 bytes) 000402 1763351353.172618 mapread 0x001555 thru 0x00aef1 (0x999d bytes) 000403 1763351353.173685 trunc from 0x01873a to 0x03aad0 (0x22397 bytes) 000406 1763351353.195481 trunc from 0x03aad1 to 0x007192 (0x3393e bytes) 000407 1763351353.222485 read 0x006e50 thru 0x007192 (0x343 bytes) 000408 1763351353.223348 read 0x001295 thru 0x002dc5 (0x1b31 bytes) 000409 1763351353.223833 mapwrite 0x03753e thru 0x03be83 (0x4946 bytes) 000411 1763351353.315188 read 0x03aba9 thru 0x03be83 (0x12db bytes) 000412 1763351353.315727 trunc from 0x03be84 to 0x014008 (0x27e7b bytes) 000414 1763351353.343323 trunc from 0x014009 to 0x00049c (0x13b6c bytes) 000415 1763351353.369052 mapread 0x000042 thru 0x00049c (0x45b bytes) 000416 1763351353.370609 trunc from 0x00049d to 0x01605b (0x15bbf bytes) 000419 1763351353.393156 write 0x00f1bf thru 0x019c44 (0xaa86 bytes) 000421 1763351353.404723 read 0x017742 thru 0x019c44 (0x2503 bytes) 000422 1763351353.405391 read 0x00f493 thru 0x0119a4 (0x2512 bytes) 000423 1763351353.416752 MIRROR_EXTEND to 2 mirrors 000424 1763351354.035971 write 0x0378a5 thru 0x03ffff (0x875b bytes) 000425 1763351354.107504 read 0x0132fa thru 0x0166d6 (0x33dd bytes) 000428 1763351354.123498 mapread 0x033d19 thru 0x033f74 (0x25c bytes) 000429 1763351354.134212 mapread 0x02fd7f thru 0x038adf (0x8d61 bytes) 000430 1763351354.151195 mapwrite 0x016655 thru 0x024e40 (0xe7ec bytes) 000431 1763351354.252857 read 0x01f509 thru 0x02557c (0x6074 bytes) 000432 1763351354.254837 write 0x009cb8 thru 0x013f97 (0xa2e0 bytes) 000433 1763351354.277101 mapwrite 0x020ac9 thru 0x022176 (0x16ae bytes) 000437 1763351354.382382 mapread 0x031042 thru 0x0387da (0x7799 bytes) 000438 1763351354.383617 read 0x00923e thru 0x00b469 (0x222c bytes) 000439 1763351354.384209 read 0x01bcaa thru 0x026a64 (0xadbb bytes) 000440 1763351354.405302 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000443 1763351354.622299 MIRROR_EXTEND to 2 mirrors 000445 1763351355.398085 read 0x001c4d thru 0x0083ea (0x679e bytes) 000447 1763351355.410466 trunc from 0x040000 to 0x039c3a (0x63c5 bytes) 000448 1763351355.524556 mapread 0x00e251 thru 0x00ef44 (0xcf4 bytes) 000449 1763351355.532153 mapwrite 0x026366 thru 0x035d84 (0xfa1f bytes) 000450 1763351355.607233 mapread 0x001693 thru 0x005636 (0x3fa4 bytes) 000453 1763351355.622773 MIRROR_RESYNC 2 mirrors 000454 1763351356.329205 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000455 1763351356.465074 mapwrite 0x0071ad thru 0x01528a (0xe0de bytes) 000456 1763351356.661446 mapwrite 0x03ca09 thru 0x03ffff (0x35f7 bytes) 000457 1763351356.825226 trunc from 0x040000 to 0x020664 (0x1f99b bytes) 000458 1763351356.848790 write 0x00e6f4 thru 0x015cdd (0x75ea bytes) 000461 1763351356.852794 trunc from 0x020665 to 0x03a072 (0x19a0e bytes) 000462 1763351356.874930 trunc from 0x03a073 to 0x02495f (0x15713 bytes) 000466 1763351356.907699 MIRROR_EXTEND to 2 mirrors 000467 1763351357.622522 mapwrite 0x039a85 thru 0x03ffff (0x657b bytes) 000468 1763351357.814338 write 0x02e7d2 thru 0x038dff (0xa62e bytes) 000469 1763351357.836205 trunc from 0x040000 to 0x000a7e (0x3f581 bytes) 000470 1763351357.867579 trunc from 0x000a7f to 0x0384a6 (0x37a28 bytes) 000474 1763351357.895348 mapwrite 0x005bca thru 0x00b444 (0x587b bytes) 000475 1763351357.983219 read 0x036077 thru 0x0384a6 (0x2430 bytes) 000477 1763351358.002078 trunc from 0x0384a7 to 0x02fba5 (0x8901 bytes) 000478 1763351358.035120 mapread 0x00f313 thru 0x01cd49 (0xda37 bytes) 000479 1763351358.102904 read 0x00052f thru 0x005ae1 (0x55b3 bytes) 000481 1763351358.132154 trunc from 0x02fba6 to 0x0072a6 (0x288ff bytes) 000482 1763351358.183338 mapwrite 0x03693e thru 0x03b029 (0x46ec bytes) 000483 1763351358.340398 trunc from 0x03b02a to 0x006aa5 (0x34584 bytes) 000484 1763351358.367299 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000485 1763351358.534622 mapread 0x0008f9 thru 0x00603c (0x5744 bytes) 000487 1763351358.556155 mapwrite 0x034d24 thru 0x03ffff (0xb2dc bytes) 000488 1763351358.638827 write 0x03e355 thru 0x03ffff (0x1cab bytes) 000491 1763351358.644764 mapwrite 0x00b844 thru 0x00d61b (0x1dd8 bytes) 000492 1763351358.719932 trunc from 0x040000 to 0x010598 (0x2fa67 bytes) 000494 1763351358.740219 read 0x00246e thru 0x003f96 (0x1b29 bytes) 000496 1763351358.749652 write 0x026d27 thru 0x02d136 (0x6410 bytes) 000497 1763351358.751159 read 0x008490 thru 0x00ddb0 (0x5921 bytes) 000498 1763351358.751776 mapwrite 0x03be80 thru 0x03f132 (0x32b3 bytes) 000504 1763351358.880636 mapwrite 0x025c2f thru 0x02c590 (0x6962 bytes) 000506 1763351358.922301 read 0x017d5b thru 0x0237e7 (0xba8d bytes) 000508 1763351358.924577 write 0x00f2d8 thru 0x015d70 (0x6a99 bytes) 000510 1763351358.932922 write 0x00bf14 thru 0x013572 (0x765f bytes) 000512 1763351358.944480 read 0x03abe5 thru 0x03f132 (0x454e bytes) 000513 1763351358.950638 trunc from 0x03f133 to 0x003337 (0x3bdfb bytes) 000514 1763351358.996032 trunc from 0x003338 to 0x017bd8 (0x148a1 bytes) 000516 1763351359.016287 trunc from 0x017bd9 to 0x0358d3 (0x1dcfb bytes) 000518 1763351359.039388 read 0x00b408 thru 0x00dc67 (0x2860 bytes) 000520 1763351359.051386 write 0x0220b4 thru 0x022210 (0x15d bytes) 000521 1763351359.053207 read 0x029a4e thru 0x033790 (0x9d43 bytes) 000522 1763351359.055508 mapread 0x0095a9 thru 0x00aafe (0x1556 bytes) 000524 1763351359.057217 read 0x01850e thru 0x025540 (0xd033 bytes) 000525 1763351359.058465 read 0x022551 thru 0x02cda8 (0xa858 bytes) 000526 1763351359.059536 read 0x02efd4 thru 0x0318aa (0x28d7 bytes) 000538 1763351359.066774 mapwrite 0x033275 thru 0x0364e7 (0x3273 bytes) 000540 1763351359.164070 read 0x018371 thru 0x01dd82 (0x5a12 bytes) 000542 1763351359.165221 write 0x0161dd thru 0x01a677 (0x449b bytes) 000544 1763351359.171532 write 0x020ad8 thru 0x02c745 (0xbc6e bytes) 000548 1763351359.181415 MIRROR_EXTEND to 2 mirrors 000551 1763351359.820513 read 0x0255ef thru 0x0280f7 (0x2b09 bytes) 000554 1763351359.836362 write 0x021ab8 thru 0x023c90 (0x21d9 bytes) 000557 1763351359.929415 trunc from 0x0364e8 to 0x03802b (0x1b44 bytes) 000558 1763351359.980238 write 0x001ff8 thru 0x00522b (0x3234 bytes) 000559 1763351359.991854 read 0x0305d3 thru 0x03802b (0x7a59 bytes) 000560 1763351360.005306 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000564 1763351360.189182 write 0x03b362 thru 0x03ffff (0x4c9e bytes) 000565 1763351360.193014 mapread 0x010232 thru 0x01dd1f (0xdaee bytes) 000566 1763351360.229267 read 0x00d777 thru 0x01682d (0x90b7 bytes) 000568 1763351360.230752 write 0x01893f thru 0x023802 (0xaec4 bytes) 000571 1763351360.236802 trunc from 0x040000 to 0x015c11 (0x2a3ee bytes) 000575 1763351360.287438 read 0x00d4f1 thru 0x015c11 (0x8721 bytes) 000577 1763351360.288569 read 0x010b50 thru 0x015c11 (0x50c2 bytes) 000579 1763351360.290350 trunc from 0x015c12 to 0x03baf8 (0x25ee7 bytes) 000581 1763351360.302802 mapread 0x039fdc thru 0x03b04e (0x1073 bytes) 000582 1763351360.316753 MIRROR_EXTEND to 2 mirrors 000583 1763351360.899488 mapread 0x02d7ee thru 0x03baf8 (0xe30b bytes) 000585 1763351360.923570 read 0x00b01e thru 0x017f26 (0xcf09 bytes) 000586 1763351360.930366 trunc from 0x03baf9 to 0x036a90 (0x5068 bytes) 000591 1763351361.016381 MIRROR_RESYNC 2 mirrors 000592 1763351361.658692 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000593 1763351361.847854 write 0x00581d thru 0x00e293 (0x8a77 bytes) 000596 1763351361.876955 write 0x01ec5f thru 0x024d43 (0x60e5 bytes) 000597 1763351361.888682 read 0x0358e7 thru 0x036a90 (0x11aa bytes) 000598 1763351361.893985 read 0x02dc63 thru 0x031b27 (0x3ec5 bytes) 000601 1763351361.908010 mapwrite 0x00ad48 thru 0x00fe2b (0x50e4 bytes) 000604 1763351361.970120 mapwrite 0x00e934 thru 0x015692 (0x6d5f bytes) 000606 1763351362.051783 trunc from 0x036a91 to 0x02dfeb (0x8aa5 bytes) 000607 1763351362.073939 mapread 0x018478 thru 0x01b277 (0x2e00 bytes) 000609 1763351362.079292 mapread 0x02c14a thru 0x02dfeb (0x1ea2 bytes) 000611 1763351362.091409 MIRROR_EXTEND to 2 mirrors 000612 1763351362.738722 MIRROR_EXTEND to 3 mirrors 000613 1763351363.386315 mapwrite 0x026fa5 thru 0x02ca2a (0x5a86 bytes) 000614 1763351363.560032 mapread 0x008477 thru 0x010c73 (0x87fd bytes) 000615 1763351363.570900 mapread 0x000f87 thru 0x0065ad (0x5627 bytes) 000618 1763351363.582663 read 0x01f624 thru 0x02b975 (0xc352 bytes) 000620 1763351363.594751 write 0x02bae6 thru 0x02d74c (0x1c67 bytes) 000621 1763351363.595794 read 0x000576 thru 0x00b963 (0xb3ee bytes) 000622 1763351363.596459 mapread 0x017893 thru 0x027006 (0xf774 bytes) 000623 1763351363.607180 write 0x0145c5 thru 0x023956 (0xf392 bytes) 000624 1763351363.612493 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x3f6bb 000626 1763351364.382407 trunc from 0x02dfec to 0x03f6ba (0x116cf bytes) 000628 1763351364.454286 read 0x0089a1 thru 0x00e389 (0x59e9 bytes) 000629 1763351364.463270 read 0x03c0d4 thru 0x03f6ba (0x35e7 bytes) 000631 1763351364.478624 mapwrite 0x01d2b2 thru 0x028504 (0xb253 bytes) 000633 1763351364.558305 read 0x03849c thru 0x03be4c (0x39b1 bytes) 000634 1763351364.558993 mapwrite 0x01b217 thru 0x01efb3 (0x3d9d bytes) 000635 1763351364.606711 mapread 0x02a640 thru 0x03550d (0xaece bytes) 000638 1763351364.617703 MIRROR_RESYNC 3 mirrors 000639 1763351365.259141 mapread 0x01b44e thru 0x01b9c4 (0x577 bytes) 000640 1763351365.273420 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000642 1763351365.595672 read 0x00849a thru 0x00c4ca (0x4031 bytes) 000645 1763351365.606566 MIRROR_EXTEND to 3 mirrors 000646 1763351366.309669 mapread 0x03db8c thru 0x03f6ba (0x1b2f bytes) 000647 1763351366.377459 MIRROR_EXTEND to 4 mirrors 000648 1763351367.171396 mapwrite 0x020e94 thru 0x030251 (0xf3be bytes) 000650 1763351367.423020 trunc from 0x03f6bb to 0x012376 (0x2d344 bytes) 000654 1763351367.465646 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x12377): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12377): Mirror 2: 0x3543f961 (stale) Mirror 3: 0x3543f961 (stale) Mirror 4: 0x3543f961 (stale) CRC-32 checksum value for chunk [0, 0x12377): Mirror 1: 0x3543f961 Mirror 5: 0x3543f961 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12377: skipped 000655 1763351368.562976 mapwrite 0x00d3a2 thru 0x01b5d5 (0xe234 bytes) 000656 1763351368.727484 write 0x02e850 thru 0x031e4a (0x35fb bytes) 000657 1763351368.738598 MIRROR_RESYNC 5 mirrors 000658 1763351369.801288 read 0x0222ed thru 0x022313 (0x27 bytes) 000662 1763351369.808170 mapread 0x02b229 thru 0x031e4a (0x6c22 bytes) 000664 1763351369.828656 write 0x03a024 thru 0x03ae2c (0xe09 bytes) 000665 1763351369.892815 write 0x03b064 thru 0x03fa59 (0x49f6 bytes) 000668 1763351369.896058 mapwrite 0x02fc59 thru 0x03e464 (0xe80c bytes) 000669 1763351369.962621 mapwrite 0x00be67 thru 0x016877 (0xaa11 bytes) 000670 1763351370.060830 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3fa5a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa5a): Mirror 2: 0x68a9ae18 (stale) Mirror 3: 0x68a9ae18 (stale) Mirror 4: 0x68a9ae18 (stale) Mirror 5: 0x68a9ae18 (stale) CRC-32 checksum value for chunk [0, 0x3fa5a): Mirror 1: 0x2043eb69 Mirror 6: 0x2043eb69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa5a: skipped 000671 1763351371.145300 trunc from 0x03fa5a to 0x037d3b (0x7d1e bytes) 000674 1763351371.296588 mapread 0x00367a thru 0x00db36 (0xa4bd bytes) 000675 1763351371.319684 mapread 0x01ad5a thru 0x0241d3 (0x947a bytes) 000677 1763351371.345092 read 0x01a9c5 thru 0x0201b3 (0x57ef bytes) 000678 1763351371.346847 trunc from 0x037d3c to 0x02f916 (0x8425 bytes) 000680 1763351371.378956 mapread 0x028def thru 0x02f84a (0x6a5c bytes) 000682 1763351371.388946 write 0x005755 thru 0x009be1 (0x448d bytes) 000683 1763351371.390005 mapwrite 0x026a2f thru 0x03478e (0xdd60 bytes) 000684 1763351371.507803 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3478f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3478f): Mirror 2: 0x7329f651 (stale) Mirror 3: 0x7329f651 (stale) Mirror 4: 0x7329f651 (stale) Mirror 5: 0x7329f651 (stale) Mirror 6: 0x416cbf28 (stale) CRC-32 checksum value for chunk [0, 0x3478f): Mirror 1: 0xa7573986 Mirror 7: 0xa7573986 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3478f: skipped 000685 1763351372.608856 trunc from 0x03478f to 0x00e176 (0x26618 bytes) 000686 1763351372.743227 mapread 0x00981a thru 0x00e176 (0x495d bytes) 000687 1763351372.762574 trunc from 0x00e177 to 0x00f452 (0x12dc bytes) 000688 1763351372.780250 read 0x00ee1b thru 0x00f452 (0x638 bytes) 000689 1763351372.790036 mapwrite 0x033280 thru 0x03354f (0x2d0 bytes) 000691 1763351372.872695 write 0x02b401 thru 0x032632 (0x7232 bytes) 000692 1763351372.883007 write 0x0296d6 thru 0x031113 (0x7a3e bytes) 000693 1763351372.884194 read 0x027fe5 thru 0x03354f (0xb56b bytes) 000694 1763351372.885496 mapwrite 0x00e538 thru 0x00f811 (0x12da bytes) 000696 1763351373.003232 mapread 0x005fd1 thru 0x00a643 (0x4673 bytes) 000698 1763351373.019133 MIRROR_RESYNC 7 mirrors 000701 1763351374.444077 write 0x022ff2 thru 0x02dba2 (0xabb1 bytes) 000703 1763351374.570417 mapread 0x014228 thru 0x015724 (0x14fd bytes) 000704 1763351374.587574 read 0x0302ff thru 0x03354f (0x3251 bytes) 000705 1763351374.594120 mapread 0x008f55 thru 0x009b69 (0xc15 bytes) 000707 1763351374.613189 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33550): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33550): Mirror 2: 0xcbfc256f (stale) Mirror 3: 0xcbfc256f (stale) Mirror 4: 0xcbfc256f (stale) Mirror 5: 0xcbfc256f (stale) Mirror 6: 0xcbfc256f (stale) Mirror 7: 0xcbfc256f (stale) CRC-32 checksum value for chunk [0, 0x33550): Mirror 1: 0x7734be98 Mirror 8: 0x7734be98 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33550: skipped 000708 1763351375.625783 read 0x028534 thru 0x02fa26 (0x74f3 bytes) 000711 1763351375.641166 read 0x02c9eb thru 0x03354f (0x6b65 bytes) 000712 1763351375.658276 mapwrite 0x0090fd thru 0x00b8a0 (0x27a4 bytes) 000713 1763351375.872736 trunc from 0x033550 to 0x00c752 (0x26dfd bytes) 000716 1763351375.930862 write 0x00a01c thru 0x017e6e (0xde53 bytes) 000717 1763351375.932343 mapread 0x008953 thru 0x00c908 (0x3fb6 bytes) 000718 1763351375.942750 write 0x001c6d thru 0x0065e4 (0x4978 bytes) 000719 1763351375.958173 mapread 0x0142ca thru 0x017e6e (0x3ba5 bytes) 000720 1763351375.958987 mapread 0x0026af thru 0x00a785 (0x80d7 bytes) 000722 1763351375.964204 write 0x02b322 thru 0x02e466 (0x3145 bytes) 000724 1763351375.966302 write 0x0113a4 thru 0x01d150 (0xbdad bytes) 000725 1763351375.978725 MIRROR_RESYNC 8 mirrors 000726 1763351377.424320 trunc from 0x02e467 to 0x00c00d (0x22459 bytes) 000728 1763351377.567675 write 0x0395b3 thru 0x03ffff (0x6a4d bytes) 000729 1763351377.568998 mapread 0x0382ac thru 0x03ffff (0x7d54 bytes) 000731 1763351377.593984 write 0x000aa6 thru 0x00dfc8 (0xd523 bytes) 000732 1763351377.615463 trunc from 0x040000 to 0x01d8a7 (0x22758 bytes) 000737 1763351377.650990 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d8a8): Mirror 2: 0x634bb93d (stale) Mirror 3: 0x634bb93d (stale) Mirror 4: 0x634bb93d (stale) Mirror 5: 0x634bb93d (stale) Mirror 6: 0x634bb93d (stale) Mirror 7: 0x634bb93d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d8a8): Mirror 2: 0x634bb93d (stale) Mirror 3: 0x634bb93d (stale) Mirror 4: 0x634bb93d (stale) Mirror 5: 0x634bb93d (stale) Mirror 6: 0x634bb93d (stale) Mirror 7: 0x634bb93d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d8a8: skipped 000738 1763351378.085021 read 0x016b6a thru 0x01d8a7 (0x6d3e bytes) 000739 1763351378.093804 mapread 0x00d7ce thru 0x01c229 (0xea5c bytes) 000740 1763351378.114369 trunc from 0x01d8a8 to 0x034936 (0x1708f bytes) 000741 1763351378.134327 read 0x009b46 thru 0x00e32c (0x47e7 bytes) 000742 1763351378.135004 write 0x007cc4 thru 0x00bfb8 (0x42f5 bytes) 000744 1763351378.149347 trunc from 0x034937 to 0x02ec50 (0x5ce6 bytes) 000748 1763351378.182037 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ec51): Mirror 3: 0x81c314f9 (stale) Mirror 4: 0x81c314f9 (stale) Mirror 5: 0x81c314f9 (stale) Mirror 6: 0x81c314f9 (stale) Mirror 7: 0x81c314f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ec51): Mirror 3: 0x81c314f9 (stale) Mirror 4: 0x81c314f9 (stale) Mirror 5: 0x81c314f9 (stale) Mirror 6: 0x81c314f9 (stale) Mirror 7: 0x81c314f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ec51: skipped 000749 1763351378.649040 trunc from 0x02ec51 to 0x021f19 (0xcd37 bytes) 000752 1763351378.683230 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x21f1a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21f1a): Mirror 3: 0x3a236a7b (stale) Mirror 4: 0x3a236a7b (stale) Mirror 5: 0x3a236a7b (stale) Mirror 6: 0x3a236a7b (stale) Mirror 7: 0x3a236a7b (stale) CRC-32 checksum value for chunk [0, 0x21f1a): Mirror 1: 0x6efb7f00 Mirror 8: 0x6efb7f00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21f1a: skipped 000753 1763351379.525513 trunc from 0x021f1a to 0x01e80d (0x370c bytes) 000754 1763351379.630102 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e80e): Mirror 4: 0x21e84554 (stale) Mirror 5: 0x21e84554 (stale) Mirror 6: 0x21e84554 (stale) Mirror 7: 0x21e84554 (stale) Mirror 8: 0xf83c02d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e80e): Mirror 4: 0x21e84554 (stale) Mirror 5: 0x21e84554 (stale) Mirror 6: 0x21e84554 (stale) Mirror 7: 0x21e84554 (stale) Mirror 8: 0xf83c02d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e80e: skipped 000758 1763351380.010094 trunc from 0x01e80e to 0x010a8b (0xdd82 bytes) 000759 1763351380.025455 trunc from 0x010a8c to 0x029fa4 (0x19519 bytes) 000761 1763351380.056046 read 0x00b7eb thru 0x015b5c (0xa372 bytes) 000766 1763351380.071749 write 0x01d602 thru 0x029d0c (0xc70b bytes) 000767 1763351380.095407 read 0x013e5f thru 0x01f1bf (0xb361 bytes) 000768 1763351380.108524 mapwrite 0x0228e3 thru 0x0320e5 (0xf803 bytes) 000769 1763351380.215164 mapwrite 0x01d27a thru 0x0205f2 (0x3379 bytes) 000770 1763351380.306264 read 0x01c97e thru 0x027673 (0xacf6 bytes) 000772 1763351380.312961 MIRROR_RESYNC 6 mirrors 000774 1763351381.376787 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000776 1763351381.734480 mapwrite 0x011ffc thru 0x0198c1 (0x78c6 bytes) 000777 1763351381.900725 mapread 0x02a8f6 thru 0x02ed02 (0x440d bytes) 000779 1763351381.914306 trunc from 0x0320e6 to 0x002917 (0x2f7ce bytes) 000781 1763351381.984005 write 0x028763 thru 0x035779 (0xd017 bytes) 000784 1763351381.989706 mapread 0x005aae thru 0x011acd (0xc020 bytes) 000787 1763351382.017126 read 0x0087a9 thru 0x0161ba (0xda12 bytes) 000788 1763351382.017816 trunc from 0x03577a to 0x01f7ae (0x15fcb bytes) 000789 1763351382.047892 mapwrite 0x02fa0f thru 0x03e2a1 (0xe893 bytes) 000790 1763351382.149707 read 0x00bcb2 thru 0x016874 (0xabc3 bytes) 000793 1763351382.151809 trunc from 0x03e2a2 to 0x0287d0 (0x15ad1 bytes) 000794 1763351382.176193 mapread 0x012cb0 thru 0x015865 (0x2bb6 bytes) 000795 1763351382.177594 mapwrite 0x0295d0 thru 0x02c650 (0x3081 bytes) 000798 1763351382.274171 mapwrite 0x006246 thru 0x00d655 (0x7410 bytes) 000800 1763351382.341089 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c651): Mirror 5: 0x9de1b865 (stale) Mirror 6: 0x9de1b865 (stale) Mirror 8: 0x9de1b865 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c651): Mirror 5: 0x9de1b865 (stale) Mirror 6: 0x9de1b865 (stale) Mirror 8: 0x9de1b865 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c651: skipped 000802 1763351382.693974 trunc from 0x02c651 to 0x02598c (0x6cc4 bytes) 000803 1763351382.709900 trunc from 0x02598d to 0x03030c (0xa980 bytes) 000805 1763351382.736398 mapwrite 0x00d000 thru 0x01425a (0x725b bytes) 000806 1763351382.832924 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3030d): Mirror 5: 0x89f6752 (stale) Mirror 6: 0x89f6752 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3030d): Mirror 5: 0x89f6752 (stale) Mirror 6: 0x89f6752 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3030d: skipped 000807 1763351383.129333 mapread 0x00e089 thru 0x014c72 (0x6bea bytes) 000809 1763351383.148682 write 0x03c857 thru 0x03ffff (0x37a9 bytes) 000810 1763351383.153289 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x65e633a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x65e633a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000812 1763351383.470800 MIRROR_RESYNC 2 mirrors 000814 1763351384.098822 mapread 0x0151d0 thru 0x01e0dd (0x8f0e bytes) 000815 1763351384.119094 mapwrite 0x02ca0a thru 0x034b32 (0x8129 bytes) 000817 1763351384.297072 mapwrite 0x01cf47 thru 0x028bc9 (0xbc83 bytes) 000818 1763351384.391793 read 0x0111ee thru 0x0194b0 (0x82c3 bytes) 000819 1763351384.402980 mapwrite 0x009f36 thru 0x0100f2 (0x61bd bytes) 000820 1763351384.471675 write 0x027306 thru 0x033eb8 (0xcbb3 bytes) 000821 1763351384.474502 write 0x03dadc thru 0x03ffff (0x2524 bytes) 000824 1763351384.484764 trunc from 0x040000 to 0x02d5b5 (0x12a4a bytes) 000825 1763351384.513638 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d5b6): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d5b6): Mirror 6: 0x29370bc6 (stale) CRC-32 checksum value for chunk [0, 0x2d5b6): Mirror 1: 0xaffbc17a Mirror 7: 0xaffbc17a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d5b6: skipped 000826 1763351385.444810 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d5b6): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d5b6): Mirror 6: 0x29370bc6 (stale) CRC-32 checksum value for chunk [0, 0x2d5b6): Mirror 1: 0xaffbc17a Mirror 7: 0xaffbc17a Mirror 8: 0xaffbc17a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d5b6: skipped 000827 1763351386.375431 MIRROR_RESYNC 4 mirrors 000828 1763351387.044676 mapwrite 0x014cf6 thru 0x022728 (0xda33 bytes) 000829 1763351387.319834 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d5b6): Mirror 7: 0xaffbc17a (stale) Mirror 8: 0xaffbc17a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d5b6): Mirror 7: 0xaffbc17a (stale) Mirror 8: 0xaffbc17a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d5b6: skipped 000831 1763351387.631709 mapread 0x021e67 thru 0x02d5b5 (0xb74f bytes) 000832 1763351387.660609 trunc from 0x02d5b6 to 0x010f3f (0x1c676 bytes) 000835 1763351387.681628 write 0x022d3f thru 0x02d0f9 (0xa3bb bytes) 000836 1763351387.688722 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d0fa): Mirror 8: 0x1af6aea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0fa): Mirror 8: 0x1af6aea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0fa: skipped 000838 1763351387.992464 read 0x01d5ce thru 0x027f11 (0xa944 bytes) 000840 1763351388.006778 mapread 0x02716a thru 0x02d0f9 (0x5f90 bytes) 000841 1763351388.030121 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) 000842 1763351388.181788 trunc from 0x02d0fa to 0x011927 (0x1b7d2 bytes) 000843 1763351388.195713 read 0x003fb6 thru 0x00be20 (0x7e6b bytes) 000845 1763351388.204439 mapread 0x0018cc thru 0x0087b9 (0x6eee bytes) 000846 1763351388.212378 mapwrite 0x02759e thru 0x0336df (0xc142 bytes) 000849 1763351388.298996 MIRROR_EXTEND to 2 mirrors 000850 1763351388.819415 read 0x03114e thru 0x0326ba (0x156d bytes) 000851 1763351388.828010 mapread 0x02a960 thru 0x0336df (0x8d80 bytes) 000854 1763351388.850630 trunc from 0x0336e0 to 0x01d4e4 (0x161fb bytes) 000855 1763351388.972472 mapwrite 0x01afdb thru 0x02388a (0x88b0 bytes) 000856 1763351389.061282 mapread 0x012891 thru 0x01a436 (0x7ba6 bytes) 000857 1763351389.070771 mapread 0x013e27 thru 0x01a357 (0x6531 bytes) 000862 1763351389.079753 MIRROR_RESYNC 2 mirrors 000863 1763351389.655785 read 0x000a7c thru 0x004b84 (0x4109 bytes) 000864 1763351389.675310 MIRROR_EXTEND to 3 mirrors 000866 1763351390.472348 mapwrite 0x02955c thru 0x02cda9 (0x384e bytes) 000867 1763351390.639011 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2cdaa): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cdaa): Mirror 2: 0x461e7503 (stale) Mirror 3: 0x461e7503 (stale) CRC-32 checksum value for chunk [0, 0x2cdaa): Mirror 1: 0x7bd08a8b Mirror 4: 0x7bd08a8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cdaa: skipped 000868 1763351391.760097 mapread 0x01a2bf thru 0x02601a (0xbd5c bytes) 000870 1763351391.793250 mapwrite 0x03c5c6 thru 0x03ffff (0x3a3a bytes) 000873 1763351392.069706 mapread 0x01e1cf thru 0x02621a (0x804c bytes) 000874 1763351392.114356 trunc from 0x040000 to 0x01e3e8 (0x21c17 bytes) 000875 1763351392.162605 mapread 0x00d9b5 thru 0x01d81b (0xfe67 bytes) 000876 1763351392.181284 mapread 0x00360a thru 0x005672 (0x2069 bytes) 000877 1763351392.192788 mapread 0x00ae60 thru 0x013d13 (0x8eb4 bytes) 000878 1763351392.193710 read 0x00ae32 thru 0x00fdcb (0x4f9a bytes) 000883 1763351392.196447 write 0x008888 thru 0x00fdbd (0x7536 bytes) 000886 1763351392.199708 read 0x0058e3 thru 0x0126a6 (0xcdc4 bytes) 000887 1763351392.201493 mapwrite 0x0188a2 thru 0x01d694 (0x4df3 bytes) 000888 1763351392.257325 read 0x0014f2 thru 0x006c09 (0x5718 bytes) 000890 1763351392.258527 trunc from 0x01e3e9 to 0x0183ce (0x601a bytes) 000891 1763351392.283667 mapread 0x017549 thru 0x0183ce (0xe86 bytes) 000892 1763351392.292485 MIRROR_RESYNC 4 mirrors 000893 1763351393.137674 mapwrite 0x00db6c thru 0x01762b (0x9ac0 bytes) 000895 1763351393.311979 MIRROR_RESYNC 4 mirrors 000896 1763351394.042840 trunc from 0x0183cf to 0x036b7b (0x1e7ad bytes) 000898 1763351394.134595 trunc from 0x036b7c to 0x038bd5 (0x205a bytes) 000899 1763351394.155966 write 0x008b6f thru 0x00921c (0x6ae bytes) 000900 1763351394.177628 read 0x0282f9 thru 0x02fc3c (0x7944 bytes) 000901 1763351394.196099 read 0x0094d6 thru 0x0116c1 (0x81ec bytes) 000902 1763351394.212652 write 0x00157d thru 0x0097d9 (0x825d bytes) 000903 1763351394.218672 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38bd6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bd6): Mirror 2: 0x3f7eb95d (stale) Mirror 3: 0x3f7eb95d (stale) Mirror 4: 0x3f7eb95d (stale) CRC-32 checksum value for chunk [0, 0x38bd6): Mirror 1: 0x95c64cb4 Mirror 5: 0x95c64cb4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bd6: skipped 000904 1763351394.980776 MIRROR_RESYNC 5 mirrors 000906 1763351395.948564 write 0x01a85d thru 0x025273 (0xaa17 bytes) 000908 1763351396.029539 mapread 0x0351c0 thru 0x0353f2 (0x233 bytes) 000909 1763351396.045044 mapread 0x013a3f thru 0x01f424 (0xb9e6 bytes) 000912 1763351396.056365 read 0x002337 thru 0x01053b (0xe205 bytes) 000913 1763351396.064226 trunc from 0x038bd6 to 0x00dd1d (0x2aeb8 bytes) 000915 1763351396.123641 mapread 0x000e7a thru 0x00cf16 (0xc09d bytes) 000917 1763351396.147033 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdd1e): Mirror 2: 0x1bac65b1 (stale) Mirror 4: 0x1bac65b1 (stale) Mirror 5: 0x1bac65b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdd1e): Mirror 2: 0x1bac65b1 (stale) Mirror 4: 0x1bac65b1 (stale) Mirror 5: 0x1bac65b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdd1e: skipped 000918 1763351396.605644 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdd1e): Mirror 2: 0x1bac65b1 (stale) Mirror 5: 0x1bac65b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdd1e): Mirror 2: 0x1bac65b1 (stale) Mirror 5: 0x1bac65b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdd1e: skipped 000920 1763351396.899120 mapread 0x004736 thru 0x00dd1d (0x95e8 bytes) 000924 1763351396.920424 read 0x0063d6 thru 0x00ce33 (0x6a5e bytes) 000925 1763351396.921001 mapread 0x00b65c thru 0x00dd1d (0x26c2 bytes) 000927 1763351396.927238 MIRROR_RESYNC 3 mirrors 000928 1763351397.685502 mapread 0x004831 thru 0x00c4df (0x7caf bytes) 000929 1763351397.719181 trunc from 0x00dd1e to 0x03c0e2 (0x2e3c5 bytes) 000933 1763351397.815525 read 0x0329c8 thru 0x03392f (0xf68 bytes) 000934 1763351397.823731 mapwrite 0x019878 thru 0x020a16 (0x719f bytes) 000935 1763351397.890394 write 0x02a8ae thru 0x02b875 (0xfc8 bytes) 000936 1763351397.899555 mapread 0x029ea4 thru 0x0360ea (0xc247 bytes) 000937 1763351397.909846 write 0x002bd8 thru 0x010839 (0xdc62 bytes) 000939 1763351397.921999 write 0x0158ae thru 0x018df1 (0x3544 bytes) 000940 1763351397.928693 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c0e3): Mirror 2: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c0e3): Mirror 2: 0xf9a23de4 (stale) Mirror 5: 0xf9a23de4 (stale) CRC-32 checksum value for chunk [0, 0x3c0e3): Mirror 1: 0x9226e7f3 Mirror 6: 0x9226e7f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c0e3: skipped 000941 1763351398.986724 mapwrite 0x009dd1 thru 0x013b80 (0x9db0 bytes) 000942 1763351399.172301 read 0x02e72a thru 0x03963a (0xaf11 bytes) 000943 1763351399.182713 mapwrite 0x007aaa thru 0x012f72 (0xb4c9 bytes) 000944 1763351399.262469 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c0e3): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c0e3): Mirror 2: 0xf9a23de4 (stale) Mirror 5: 0xf9a23de4 (stale) Mirror 6: 0x9226e7f3 (stale) CRC-32 checksum value for chunk [0, 0x3c0e3): Mirror 1: 0x8aafab5 Mirror 7: 0x8aafab5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c0e3: skipped 000945 1763351400.293130 write 0x02ef29 thru 0x0302e8 (0x13c0 bytes) 000947 1763351400.417696 mapwrite 0x0050dd thru 0x00dc3d (0x8b61 bytes) 000949 1763351400.495666 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c0e3): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c0e3): Mirror 2: 0xf9a23de4 (stale) Mirror 5: 0xf9a23de4 (stale) Mirror 6: 0x9226e7f3 (stale) Mirror 7: 0x8aafab5 (stale) CRC-32 checksum value for chunk [0, 0x3c0e3): Mirror 1: 0x5206d588 Mirror 8: 0x5206d588 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c0e3: skipped 000950 1763351401.461956 write 0x01759a thru 0x01c902 (0x5369 bytes) 000951 1763351401.594546 mapread 0x0360c1 thru 0x0378ff (0x183f bytes) 000952 1763351401.609968 trunc from 0x03c0e3 to 0x02240f (0x19cd3 bytes) 000953 1763351401.664595 trunc from 0x022410 to 0x00577a (0x1cc95 bytes) 000956 1763351401.697402 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x577b): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x577b): Mirror 2: 0xb9df55cf (stale) Mirror 5: 0xb9df55cf (stale) Mirror 6: 0x6a901286 (stale) Mirror 7: 0x6a901286 (stale) Mirror 8: 0xf61360d3 (stale) CRC-32 checksum value for chunk [0, 0x577b): Mirror 1: 0xf61360d3 Mirror 9: 0xf61360d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x577b: skipped 000957 1763351402.871245 write 0x02fdf6 thru 0x0374d3 (0x76de bytes) 000958 1763351402.995633 read 0x00586c thru 0x008f0c (0x36a1 bytes) 000959 1763351403.014334 write 0x00e8ee thru 0x0130b9 (0x47cc bytes) 000961 1763351403.046280 mapread 0x0227ae thru 0x026651 (0x3ea4 bytes) 000965 1763351403.064802 read 0x014df2 thru 0x01a025 (0x5234 bytes) 000966 1763351403.073858 read 0x01d06d thru 0x026390 (0x9324 bytes) 000967 1763351403.075437 read 0x026ef5 thru 0x03423e (0xd34a bytes) 000972 1763351403.078666 write 0x01e88a thru 0x02bf50 (0xd6c7 bytes) 000973 1763351403.080507 mapread 0x0327d9 thru 0x0374d3 (0x4cfb bytes) 000974 1763351403.092409 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x374d4): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x374d4): Mirror 2: 0xb0e68d37 (stale) Mirror 5: 0xb0e68d37 (stale) Mirror 6: 0x44c626b9 (stale) Mirror 7: 0x756d5073 (stale) Mirror 8: 0xe1704ffd (stale) Mirror 9: 0x65ec307e (stale) CRC-32 checksum value for chunk [0, 0x374d4): Mirror 1: 0x3b6aa665 Mirror 10: 0x3b6aa665 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x374d4: skipped 000975 1763351403.947833 write 0x0110c8 thru 0x01a7c5 (0x96fe bytes) 000976 1763351404.049984 mapread 0x0239e9 thru 0x031dcc (0xe3e4 bytes) 000979 1763351404.075396 trunc from 0x0374d4 to 0x033a7a (0x3a59 bytes) 000982 1763351404.121922 write 0x020503 thru 0x02f55c (0xf05a bytes) 000983 1763351404.124499 mapread 0x00072f thru 0x0020de (0x19b0 bytes) 000984 1763351404.142049 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x33a7b): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a7b): Mirror 2: 0xcfe9c31e (stale) Mirror 5: 0xcfe9c31e (stale) Mirror 6: 0x4169263f (stale) Mirror 7: 0xf0d938d6 (stale) Mirror 8: 0x58b8fef (stale) Mirror 9: 0x2aa90b5 (stale) Mirror 10: 0x6b135c89 (stale) CRC-32 checksum value for chunk [0, 0x33a7b): Mirror 1: 0x33d303c0 Mirror 11: 0x33d303c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a7b: skipped 000985 1763351405.159762 read 0x0066fd thru 0x014ef8 (0xe7fc bytes) 000986 1763351405.173283 read 0x0142a3 thru 0x016d12 (0x2a70 bytes) 000987 1763351405.187456 read 0x024403 thru 0x0268cd (0x24cb bytes) 000988 1763351405.192438 write 0x028b71 thru 0x02fe6a (0x72fa bytes) 000990 1763351405.316473 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x192:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a7b): Mirror 2: 0xcfe9c31e (stale) Mirror 5: 0xcfe9c31e (stale) Mirror 6: 0x4169263f (stale) Mirror 7: 0xf0d938d6 (stale) Mirror 8: 0x58b8fef (stale) Mirror 9: 0x2aa90b5 (stale) Mirror 10: 0x6b135c89 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a7b): Mirror 2: 0xcfe9c31e (stale) Mirror 5: 0xcfe9c31e (stale) Mirror 6: 0x4169263f (stale) Mirror 7: 0xf0d938d6 (stale) Mirror 8: 0x58b8fef (stale) Mirror 9: 0x2aa90b5 (stale) Mirror 10: 0x6b135c89 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a7b: skipped 000992 1763351405.893811 read 0x02ebb1 thru 0x02f1b6 (0x606 bytes) 000993 1763351405.903035 read 0x02493b thru 0x025901 (0xfc7 bytes) 000994 1763351405.918530 trunc from 0x033a7b to 0x01be51 (0x17c29 bytes) 000995 1763351405.999209 read 0x012122 thru 0x01be51 (0x9d30 bytes) 000997 1763351406.000830 mapwrite 0x0194ca thru 0x01ef52 (0x5a89 bytes) 000998 1763351406.073828 trunc from 0x01ef53 to 0x035708 (0x167b6 bytes) All operations completed A-OK! PASS 70a (78s) == sanity-flr test 71: check mirror extend parent fid ==== 22:50:11 (1763351411) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 200a: stress test ===================== 22:50:14 (1763351414) Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre2 Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2596096 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2754304 .. failed Extending file size to 5571808 .. Extending file size to 4780640 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5246976 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9292832 .. Extending file size to 7386112 .. Extending file size to 7413024 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7449856 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8198880 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8902656 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 54771 54772 54775 54776 54778 failed 192.168.201.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg122-client.virtnet /mnt/lustre2 (opts:) 192.168.201.122@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg122-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:51:31 (1763351491) Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre2 Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5335136 .. Extending file size to 1894400 .. failed Extending file size to 7322688 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6602176 .. Extending file size to 8888352 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8855840 .. Extending file size to 9294016 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9081536 .. Waiting 56553 56554 56555 56556 56560 done 192.168.201.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg122-client.virtnet /mnt/lustre2 (opts:) 192.168.201.122@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg122-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:52:48 (1763351568) Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0873528 s, 24.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.369417 s, 5.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3680ed91 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg122-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:53:05 (1763351585) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0770416 s, 27.2 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 22:53:13 (1763351593) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1db:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x259:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x259:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x25c:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:53:22 (1763351602) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.46047 s, 4.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:53:30 (1763351610) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.299557 s, 7.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x260:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:53:39 (1763351619) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.263301 s, 8.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x261:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x261:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x262:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:54:09 (1763351649) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg122-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115453 s, 9.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x263:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x263:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x260:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:55:03 (1763351703) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg122-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.224046 s, 9.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x265:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x261:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x262:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x265:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:55:58 (1763351758) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg122-server mds-ost sync done. Creating new pool test_204f oleg122-server: Pool lustre.test_204f created Adding targets to pool oleg122-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg122-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184617 s, 11.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x266:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x263:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x266:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg122-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg122-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg122-server: Pool lustre.test_204f destroyed PASS 204f (68s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:57:06 (1763351826) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:57:11 (1763351831) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:57:15 (1763351835) oleg122-server: Pool lustre.test_206 created Adding targets to pool oleg122-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg122-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg122-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg122-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg122-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (20s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:57:35 (1763351855) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387082 s, 27.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522717 s, 20.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xabb54f83 Mirror 3: 0xabb54f83 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:57:43 (1763351863) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:57:45 (1763351865) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:57:47 (1763351867) Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.106752 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x10b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1cd0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1cd0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1cd0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1120000 Read this (4 bytes): BBBB 192.168.201.122@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg122-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:58:15 (1763351895) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0142302 s, 576 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:58:19 (1763351899) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340414 s, 30.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x200 lmm_fid: [0x200000404:0x200:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x200 lmm_fid: [0x200000404:0x200:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x271:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834432 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 22:58:21.000000000 -0500 Modify: 2025-11-16 22:58:21.000000000 -0500 Change: 2025-11-16 22:58:21.000000000 -0500 Birth: 2025-11-16 22:58:20.000000000 -0500 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:58:25 (1763351905) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:58:27 (1763351907) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.352642 s, 116 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0507286 s, 80.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 16 22:58 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 16 22:58 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:58:32 (1763351912) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:58:37 (1763351917) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1506 sec =========== 22:58:43 (1763351923) === sanity-flr: start cleanup 22:58:44 (1763351924) === === sanity-flr: finish cleanup 22:58:48 (1763351928) ===