-----============= acceptance-small: sanity-flr ============----- Sun Nov 16 19:30:43 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg401-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:30:58 (1763339458) === oleg401-client.virtnet: executing check_config_client /mnt/lustre oleg401-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg401-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98a245336800.idle_timeout=debug osc.lustre-OST0001-osc-ffff98a245336800.idle_timeout=debug disable quota as required oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:31:22 (1763339482) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7568] == sanity-flr test 0a: lfs mirror create with -N option == 19:31:25 (1763339485) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:31:42 (1763339502) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:31:45 (1763339505) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:31:48 (1763339508) PASS 0d (18s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:32:06 (1763339526) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:32:08 (1763339528) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 19:32:11 (1763339531) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98a245336800.stats=clear osc.lustre-OST0001-osc-ffff98a245336800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:32:18 (1763339538) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:32:28 (1763339548) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:32:36 (1763339556) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:32:39 (1763339559) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 19:32:47 (1763339567) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:32:52 (1763339572) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:32:56 (1763339576) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:33:02 (1763339582) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:33:10 (1763339590) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:33:16 (1763339596) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 1.53992 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107272 s, 9.8 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:33:26 (1763339606) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0421666 s, 49.7 MB/s PASS 22 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:33:31 (1763339611) Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98a245336800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98a245336800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98a245336800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98a245336800.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg401-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (55s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:34:26 (1763339666) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.292233 s, 21.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98a245336800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98a245336800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (33s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:34:59 (1763339699) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98a245336800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98a245336800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98a245336800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98a245336800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg401-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (65s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:36:04 (1763339764) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.73777 s, 14.0 MB/s -rw-r--r-- 1 root root 52428800 Nov 16 19:36 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.953s user 0m0.001s sys 0m0.164s Stopping /mnt/lustre-ost1 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98a245336800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98a245336800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.951s user 0m0.000s sys 0m0.197s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg401-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg401-server oleg401-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98a245336800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98a245336800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.881s user 0m0.001s sys 0m0.198s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg401-server: oleg401-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg401-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg401-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98a245336800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98a245336800.ost_server_uuid in FULL state after 0 sec PASS 33b (65s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:37:09 (1763339829) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:37:12 (1763339832) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:37:15 (1763339835) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 19:37:17 (1763339837) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 19:37:24 (1763339844) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.476257 s, 26.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:37:36 (1763339856) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277422 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252712 s, 41.5 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:37:43 (1763339863) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:37:50 (1763339870) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00240611 s, 426 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000760273 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000955503 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000908978 s, 1.1 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 19:37:59 (1763339879) create test files with size 15761359 .. ef963cd7355fd878dc9920b37ff39d5f - 03fd73b78760b52c949974562df4ee7d - 743edcf42e8e8030afcfdb0538cc21c1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg401-client.virtnet /mnt/lustre (opts:) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 19:38:24 (1763339904) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:38:39 (1763339919) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:38:45 (1763339925) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124052 s, 8.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463352 s, 22.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 19:38:52 (1763339932) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174484 s, 12.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.325019 s, 12.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181316 s, 11.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 19:39:06 (1763339946) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 19:39:09 (1763339949) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:39:12 (1763339952) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165154 s, 63.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 19:39:18 (1763339958) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:39:22 (1763339962) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:39:30 (1763339970) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.2084 s, 50.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg401-client.virtnet /mnt/lustre (opts:) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:39:39 (1763339979) llite.lustre-ffff98a247433000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.157237 s, 66.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg401-client.virtnet /mnt/lustre (opts:) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff98a243931800.xattr_cache=1 PASS 44e (11s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:39:50 (1763339990) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:40:20 (1763340020) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:40:51 (1763340051) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:40:54 (1763340054) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.168465 s, 18.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.187957 s, 16.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1c90c6af (stale) Mirror 2: 0x1c90c6af (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: c88be89854ba1aa7fbff736ca302f850 - mirror 1 checksum: ec81f1c5fb3699ad3852dd1f4aa72d1a - mirror 2 checksum: c88be89854ba1aa7fbff736ca302f850 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 19:40:57 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 16 19:40:57 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c88be89854ba1aa7fbff736ca302f850 - mirror 1 checksum: ec81f1c5fb3699ad3852dd1f4aa72d1a - mirror 2 checksum: ec81f1c5fb3699ad3852dd1f4aa72d1a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 19:40:57 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 16 19:40:57 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:41:05 (1763340065) Failing mds1 on oleg401-server Stopping /mnt/lustre-mds1 (opts:) on oleg401-server 19:41:12 (1763340072) shut down facet: mds1 facet_host: oleg401-server facet_failover_host: oleg401-server Failover mds1 to oleg401-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0000 19:41:34 (1763340094) targets are mounted 19:41:34 (1763340094) facet_failover done oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798936 s, 13.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:41:51 (1763340111) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:41:54 (1763340114) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:41:57 (1763340117) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00667588 s, 2.5 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:42:03 (1763340123) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:42:06 (1763340126) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:42:09 (1763340129) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:42:29 (1763340149) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:42:57 (1763340177) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:43:19 (1763340199) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:43:45 (1763340225) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 19:43:53 (1763340233) mirror create pid 52281 mirror split pid 52282 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52281 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 52282 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 19:44:59 (1763340299) Seed set to 300 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1763340301.130888 mapwrite 0x000857 thru 0x00a3ed (0x9b97 bytes) 000002 1763340301.295907 write 0x0154cc thru 0x01e9d4 (0x9509 bytes) 000004 1763340301.297746 mapwrite 0x00c140 thru 0x0183f5 (0xc2b6 bytes) truncating to largest ever: 0x1983f 000005 1763340301.377915 trunc from 0x01e9d5 to 0x01983e (0x5196 bytes) 000006 1763340301.401052 write 0x03b4c6 thru 0x03ffff (0x4b3a bytes) 000009 1763340301.403909 read 0x01e55a thru 0x025040 (0x6ae7 bytes) 000012 1763340301.413203 read 0x01ccc1 thru 0x02179f (0x4adf bytes) 000013 1763340301.422860 write 0x00a414 thru 0x00aa4f (0x63c bytes) truncating to largest ever: 0x35ba4 000014 1763340301.423739 trunc from 0x040000 to 0x035ba3 (0xa45c bytes) 000016 1763340301.439299 write 0x03cfe8 thru 0x03ffff (0x3018 bytes) 000019 1763340301.441385 trunc from 0x040000 to 0x02b2c2 (0x14d3d bytes) truncating to largest ever: 0x391f8 000021 1763340301.470214 trunc from 0x02b2c3 to 0x0391f7 (0xdf35 bytes) 000022 1763340301.491224 mapwrite 0x01866a thru 0x01d28b (0x4c22 bytes) 000025 1763340301.529864 mapread 0x00f837 thru 0x01ced9 (0xd6a3 bytes) 000028 1763340301.532647 mapwrite 0x01c9a3 thru 0x01e5eb (0x1c49 bytes) 000029 1763340301.579030 write 0x03ca67 thru 0x03f1ef (0x2789 bytes) 000032 1763340301.583845 read 0x02dc55 thru 0x03416b (0x6517 bytes) truncating to largest ever: 0x3f7d4 000033 1763340301.614404 trunc from 0x03f1f0 to 0x03f7d3 (0x5e4 bytes) 000034 1763340301.635197 write 0x009711 thru 0x00d2b2 (0x3ba2 bytes) 000035 1763340301.636557 mapread 0x03d62f thru 0x03f7d3 (0x21a5 bytes) 000036 1763340301.645772 MIRROR_EXTEND to 2 mirrors 000037 1763340302.326279 MIRROR_RESYNC 2 mirrors 000038 1763340302.772242 MIRROR_EXTEND to 3 mirrors 000039 1763340303.448226 write 0x03885c thru 0x03b5cc (0x2d71 bytes) 000041 1763340303.568006 mapread 0x021907 thru 0x02761c (0x5d16 bytes) 000042 1763340303.597911 MIRROR_RESYNC 3 mirrors 000045 1763340304.364530 read 0x0226e4 thru 0x02fe8d (0xd7aa bytes) 000046 1763340304.374838 mapread 0x01d243 thru 0x020634 (0x33f2 bytes) 000047 1763340304.392075 mapread 0x0172b8 thru 0x01737e (0xc7 bytes) 000049 1763340304.400718 read 0x00376a thru 0x0130ac (0xf943 bytes) 000056 1763340304.417110 trunc from 0x03f7d4 to 0x005fa3 (0x39830 bytes) 000057 1763340304.507822 mapwrite 0x034375 thru 0x03ffff (0xbc8b bytes) 000059 1763340304.691744 trunc from 0x040000 to 0x010606 (0x2f9f9 bytes) 000061 1763340304.735067 write 0x01ca64 thru 0x02170c (0x4ca9 bytes) 000062 1763340304.736410 write 0x0301a1 thru 0x03d1f1 (0xd051 bytes) 000063 1763340304.756741 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d1f2): Mirror 2: 0x6fac99c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1f2): Mirror 2: 0x6fac99c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1f2: skipped 000064 1763340305.068630 read 0x0246a3 thru 0x030c8e (0xc5ec bytes) 000065 1763340305.079177 trunc from 0x03d1f2 to 0x02e5db (0xec16 bytes) 000066 1763340305.094497 trunc from 0x02e5dc to 0x014c4f (0x1998c bytes) 000067 1763340305.105172 mapwrite 0x02c990 thru 0x03afe3 (0xe654 bytes) 000068 1763340305.249066 mapread 0x02700a thru 0x02f2dd (0x82d4 bytes) 000070 1763340305.250593 write 0x0031bd thru 0x0081b4 (0x4ff8 bytes) 000071 1763340305.261000 trunc from 0x03afe4 to 0x016d15 (0x242ce bytes) 000074 1763340305.284634 mapwrite 0x024b76 thru 0x030c83 (0xc10e bytes) 000079 1763340305.380198 MIRROR_RESYNC 2 mirrors 000081 1763340305.948152 read 0x02fba1 thru 0x02ff9a (0x3fa bytes) 000083 1763340305.953345 mapwrite 0x021e9e thru 0x031d55 (0xfeb8 bytes) 000086 1763340306.150170 mapwrite 0x032918 thru 0x03ffff (0xd6e8 bytes) 000090 1763340306.284270 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) 000092 1763340306.467150 write 0x03feaa thru 0x03ffff (0x156 bytes) 000093 1763340306.474182 mapwrite 0x037af0 thru 0x03ffff (0x8510 bytes) 000096 1763340306.535965 write 0x003762 thru 0x0093fd (0x5c9c bytes) 000097 1763340306.546346 read 0x0280b7 thru 0x0356f8 (0xd642 bytes) 000098 1763340306.555849 trunc from 0x040000 to 0x00c918 (0x336e7 bytes) 000100 1763340306.579707 mapwrite 0x014156 thru 0x01d1ba (0x9065 bytes) 000102 1763340306.670694 mapread 0x002fe8 thru 0x0047af (0x17c8 bytes) 000103 1763340306.672347 read 0x01830a thru 0x01d1ba (0x4eb1 bytes) 000105 1763340306.674294 mapwrite 0x03acd4 thru 0x03b36e (0x69b bytes) 000109 1763340306.765211 write 0x0266ad thru 0x02ac33 (0x4587 bytes) 000110 1763340306.766800 write 0x034211 thru 0x0349f0 (0x7e0 bytes) 000112 1763340306.769567 trunc from 0x03b36f to 0x007e9b (0x334d3 bytes) 000113 1763340306.809639 mapwrite 0x00dfd3 thru 0x013f53 (0x5f81 bytes) 000114 1763340306.946165 write 0x02abb1 thru 0x032cf7 (0x8147 bytes) 000118 1763340306.950272 mapwrite 0x01f102 thru 0x021012 (0x1f11 bytes) 000119 1763340307.012722 write 0x005265 thru 0x0121d7 (0xcf73 bytes) 000121 1763340307.015858 write 0x039e2c thru 0x03c894 (0x2a69 bytes) 000123 1763340307.024941 MIRROR_EXTEND to 2 mirrors 000124 1763340307.805458 write 0x03b0cb thru 0x03ffff (0x4f35 bytes) 000125 1763340307.887268 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) 000127 1763340308.126474 write 0x012de8 thru 0x018a51 (0x5c6a bytes) 000129 1763340308.142358 trunc from 0x040000 to 0x03194b (0xe6b4 bytes) 000132 1763340308.162402 trunc from 0x03194c to 0x03ec02 (0xd2b7 bytes) 000134 1763340308.180990 mapread 0x0132a7 thru 0x02083b (0xd595 bytes) 000137 1763340308.200580 MIRROR_EXTEND to 2 mirrors 000138 1763340308.753251 mapwrite 0x022518 thru 0x02717c (0x4c65 bytes) 000141 1763340308.993880 read 0x03bfee thru 0x03ec02 (0x2c15 bytes) 000142 1763340309.002590 write 0x0396f8 thru 0x03c9c9 (0x32d2 bytes) 000143 1763340309.013964 read 0x03160f thru 0x03e125 (0xcb17 bytes) 000144 1763340309.031748 mapwrite 0x03f345 thru 0x03ffff (0xcbb bytes) 000149 1763340309.168079 write 0x03c6ab thru 0x03ffff (0x3955 bytes) 000150 1763340309.169394 write 0x015aac thru 0x02445a (0xe9af bytes) 000151 1763340309.177409 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdd453fd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x565ff005 Mirror 3: 0x565ff005 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000152 1763340309.868590 mapread 0x007b6b thru 0x0114d6 (0x996c bytes) 000153 1763340309.898625 write 0x015fc2 thru 0x019ff7 (0x4036 bytes) 000154 1763340310.033318 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdd453fd (stale) Mirror 3: 0x565ff005 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa80dda1a Mirror 4: 0xa80dda1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000155 1763340311.120134 read 0x010e5f thru 0x015cd1 (0x4e73 bytes) 000156 1763340311.144229 write 0x024d53 thru 0x02b61a (0x68c8 bytes) 000157 1763340311.251427 mapwrite 0x01d65e thru 0x021f56 (0x48f9 bytes) 000161 1763340311.326001 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdd453fd (stale) Mirror 3: 0x565ff005 (stale) Mirror 4: 0xa80dda1a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2d0b25b1 Mirror 5: 0x2d0b25b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000164 1763340312.147161 trunc from 0x040000 to 0x010d18 (0x2f2e7 bytes) 000165 1763340312.344843 write 0x0228c5 thru 0x02398a (0x10c6 bytes) 000167 1763340312.364374 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2398b): Mirror 3: 0xe240eb7f (stale) Mirror 4: 0xcded5e97 (stale) Mirror 5: 0xc875d44e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2398b): Mirror 3: 0xe240eb7f (stale) Mirror 4: 0xcded5e97 (stale) Mirror 5: 0xc875d44e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2398b: skipped 000168 1763340312.753449 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2398b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2398b): Mirror 3: 0xe240eb7f (stale) Mirror 4: 0xcded5e97 (stale) Mirror 5: 0xc875d44e (stale) CRC-32 checksum value for chunk [0, 0x2398b): Mirror 1: 0x1dd0e5f2 Mirror 6: 0x1dd0e5f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2398b: skipped 000172 1763340313.780592 mapwrite 0x003981 thru 0x00f8bb (0xbf3b bytes) 000173 1763340314.014049 mapread 0x018e9b thru 0x01bcb7 (0x2e1d bytes) 000176 1763340314.030100 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2398b): Mirror 4: 0xcded5e97 (stale) Mirror 5: 0xc875d44e (stale) Mirror 6: 0x1dd0e5f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2398b): Mirror 4: 0xcded5e97 (stale) Mirror 5: 0xc875d44e (stale) Mirror 6: 0x1dd0e5f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2398b: skipped 000177 1763340314.434054 trunc from 0x02398b to 0x035e65 (0x124db bytes) 000178 1763340314.466754 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35e66): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e66): Mirror 4: 0x992f6ec4 (stale) Mirror 5: 0x8c36ef76 (stale) Mirror 6: 0x450b9028 (stale) CRC-32 checksum value for chunk [0, 0x35e66): Mirror 1: 0xbf16d562 Mirror 7: 0xbf16d562 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e66: skipped 000179 1763340315.481868 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35e66): Mirror 5: 0x8c36ef76 (stale) Mirror 6: 0x450b9028 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e66): Mirror 5: 0x8c36ef76 (stale) Mirror 6: 0x450b9028 (stale) CRC-32 checksum value for chunk [0, 0x35e66): Mirror 1: 0xbf16d562 Mirror 7: 0xbf16d562 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e66: skipped 000181 1763340316.357381 write 0x02f31a thru 0x02f368 (0x4f bytes) 000182 1763340316.496116 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35e66): Mirror 6: 0x450b9028 (stale) Mirror 7: 0xbf16d562 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e66): Mirror 6: 0x450b9028 (stale) Mirror 7: 0xbf16d562 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e66: skipped 000186 1763340316.945081 mapread 0x00a5c4 thru 0x00ecae (0x46eb bytes) 000187 1763340316.987038 mapread 0x017158 thru 0x021145 (0x9fee bytes) 000189 1763340317.015676 read 0x002f7e thru 0x008633 (0x56b6 bytes) 000190 1763340317.026892 mapwrite 0x0156b0 thru 0x01a4cf (0x4e20 bytes) 000191 1763340317.093285 trunc from 0x035e66 to 0x01198e (0x244d7 bytes) 000199 1763340317.119298 trunc from 0x01198f to 0x01cfcd (0xb63f bytes) 000201 1763340317.147571 mapread 0x00c5da thru 0x01b7b4 (0xf1db bytes) 000202 1763340317.168785 MIRROR_RESYNC 3 mirrors 000203 1763340317.991999 mapwrite 0x005bec thru 0x0080fc (0x2511 bytes) 000204 1763340318.161915 mapwrite 0x00c7e1 thru 0x00d582 (0xda2 bytes) 000205 1763340318.210908 write 0x00c852 thru 0x01825d (0xba0c bytes) 000206 1763340318.216693 mapwrite 0x00ac65 thru 0x016e19 (0xc1b5 bytes) 000207 1763340318.270426 mapwrite 0x018d31 thru 0x01e65a (0x592a bytes) 000208 1763340318.362026 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e65b): Mirror 7: 0xb1854844 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e65b): Mirror 7: 0xb1854844 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e65b: skipped 000210 1763340318.639126 write 0x000b67 thru 0x0107cf (0xfc69 bytes) 000213 1763340318.653065 read 0x00abcb thru 0x0174a5 (0xc8db bytes) 000215 1763340318.663479 read 0x00542d thru 0x005904 (0x4d8 bytes) 000217 1763340318.665030 write 0x026329 thru 0x0289ca (0x26a2 bytes) 000218 1763340318.667328 mapread 0x005eb8 thru 0x00d865 (0x79ae bytes) 000219 1763340318.668857 write 0x030213 thru 0x03affc (0xadea bytes) 000221 1763340318.671438 write 0x0305a6 thru 0x030aae (0x509 bytes) 000222 1763340318.681855 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) 000223 1763340318.854006 mapread 0x02fac6 thru 0x033775 (0x3cb0 bytes) 000224 1763340318.873006 trunc from 0x03affd to 0x017f42 (0x230ba bytes) 000225 1763340318.890913 mapwrite 0x03f1ba thru 0x03ffff (0xe46 bytes) 000226 1763340319.000648 mapwrite 0x012300 thru 0x019e1f (0x7b20 bytes) 000228 1763340319.106262 mapread 0x018313 thru 0x01b8d6 (0x35c4 bytes) 000229 1763340319.119455 mapread 0x007089 thru 0x00a483 (0x33fb bytes) 000233 1763340319.135903 read 0x0392d0 thru 0x03ee5e (0x5b8f bytes) 000235 1763340319.136963 mapwrite 0x022c3b thru 0x028673 (0x5a39 bytes) 000239 1763340319.196716 trunc from 0x040000 to 0x0056ac (0x3a953 bytes) 000240 1763340319.216114 write 0x00a46e thru 0x00df2c (0x3abf bytes) 000246 1763340319.219942 write 0x03fa39 thru 0x03ffff (0x5c7 bytes) 000248 1763340319.221463 trunc from 0x040000 to 0x03b49d (0x4b62 bytes) 000249 1763340319.237037 write 0x00c00a thru 0x019a18 (0xda0f bytes) 000250 1763340319.244565 trunc from 0x03b49e to 0x00ffee (0x2b4af bytes) 000251 1763340319.263703 trunc from 0x00ffef to 0x02308d (0x1309f bytes) 000252 1763340319.283017 mapwrite 0x013e31 thru 0x01807d (0x424d bytes) 000253 1763340319.339905 read 0x00eec1 thru 0x015288 (0x63c8 bytes) 000255 1763340319.340975 mapwrite 0x002fa0 thru 0x00a863 (0x78c4 bytes) 000256 1763340319.402903 read 0x0202d5 thru 0x02308d (0x2db9 bytes) 000261 1763340319.405278 read 0x006d18 thru 0x011bc1 (0xaeaa bytes) 000263 1763340319.413055 MIRROR_EXTEND to 2 mirrors 000266 1763340320.068366 MIRROR_RESYNC 2 mirrors 000271 1763340320.369666 trunc from 0x02308e to 0x03b089 (0x17ffc bytes) 000275 1763340320.556745 read 0x020144 thru 0x02578d (0x564a bytes) 000276 1763340320.571458 trunc from 0x03b08a to 0x017660 (0x23a29 bytes) 000277 1763340320.607080 mapread 0x014508 thru 0x017660 (0x3159 bytes) 000278 1763340320.633709 read 0x009371 thru 0x016ee3 (0xdb73 bytes) 000280 1763340320.661268 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x17661): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17661): Mirror 2: 0xde537bab (stale) CRC-32 checksum value for chunk [0, 0x17661): Mirror 1: 0xde537bab Mirror 3: 0xde537bab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17661: skipped 000281 1763340321.444840 mapread 0x00e362 thru 0x017660 (0x92ff bytes) 000285 1763340321.478995 mapread 0x005b58 thru 0x0119d6 (0xbe7f bytes) 000287 1763340321.494321 mapread 0x013104 thru 0x017660 (0x455d bytes) 000288 1763340321.501307 read 0x0100b2 thru 0x015855 (0x57a4 bytes) 000289 1763340321.504891 trunc from 0x017661 to 0x0137d1 (0x3e8f bytes) 000290 1763340321.606000 mapwrite 0x0248a3 thru 0x0292a8 (0x4a06 bytes) 000292 1763340321.714302 MIRROR_RESYNC 3 mirrors 000295 1763340322.503607 mapread 0x0018d3 thru 0x0032cf (0x19fd bytes) 000296 1763340322.520111 mapread 0x02826b thru 0x0292a8 (0x103e bytes) 000299 1763340322.527898 mapread 0x027e2c thru 0x0292a8 (0x147d bytes) 000302 1763340322.529086 trunc from 0x0292a9 to 0x029221 (0x87 bytes) 000303 1763340322.610890 trunc from 0x029222 to 0x00db7f (0x1b6a2 bytes) 000307 1763340322.649632 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xdb80): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdb80): Mirror 2: 0x6c6bcfc0 (stale) Mirror 3: 0x6c6bcfc0 (stale) CRC-32 checksum value for chunk [0, 0xdb80): Mirror 1: 0x6c6bcfc0 Mirror 4: 0x6c6bcfc0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdb80: skipped 000308 1763340323.516906 mapread 0x00d738 thru 0x00db7f (0x448 bytes) 000311 1763340323.595797 mapwrite 0x005288 thru 0x0130dd (0xde56 bytes) 000312 1763340323.801478 write 0x034fa6 thru 0x037d12 (0x2d6d bytes) 000314 1763340323.803186 trunc from 0x037d13 to 0x03d29d (0x558b bytes) 000315 1763340323.836245 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d29e): Mirror 2: 0xa531f759 (stale) Mirror 3: 0xa531f759 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d29e): Mirror 2: 0xa531f759 (stale) Mirror 3: 0xa531f759 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d29e: skipped 000316 1763340324.139836 write 0x01ea60 thru 0x023208 (0x47a9 bytes) 000317 1763340324.152289 mapread 0x025106 thru 0x029408 (0x4303 bytes) 000322 1763340324.162287 read 0x027642 thru 0x031a67 (0xa426 bytes) 000323 1763340324.169599 trunc from 0x03d29e to 0x01c2a6 (0x20ff7 bytes) 000324 1763340324.193201 MIRROR_RESYNC 3 mirrors 000325 1763340325.061142 mapwrite 0x00881a thru 0x008c72 (0x459 bytes) 000326 1763340325.233143 MIRROR_RESYNC 3 mirrors 000329 1763340326.096275 read 0x0100b3 thru 0x012b10 (0x2a5e bytes) 000330 1763340326.102142 mapwrite 0x02ec11 thru 0x03d901 (0xecf1 bytes) 000331 1763340326.258196 mapread 0x02deb0 thru 0x0376d5 (0x9826 bytes) 000334 1763340326.266351 mapread 0x0073f9 thru 0x016fee (0xfbf6 bytes) 000338 1763340326.312768 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d902): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d902): Mirror 2: 0xcb9e5321 (stale) Mirror 3: 0xcb9e5321 (stale) CRC-32 checksum value for chunk [0, 0x3d902): Mirror 1: 0x5d42e4d0 Mirror 4: 0x5d42e4d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d902: skipped 000339 1763340327.026953 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d902): Mirror 3: 0xcb9e5321 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d902): Mirror 3: 0xcb9e5321 (stale) CRC-32 checksum value for chunk [0, 0x3d902): Mirror 1: 0x5d42e4d0 Mirror 4: 0x5d42e4d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d902: skipped 000341 1763340327.690042 trunc from 0x03d902 to 0x031aac (0xbe55 bytes) 000343 1763340327.837221 MIRROR_RESYNC 3 mirrors 000344 1763340328.743928 mapread 0x02fef0 thru 0x031aac (0x1bbd bytes) 000345 1763340328.763752 write 0x019ec7 thru 0x0267f1 (0xc92b bytes) 000346 1763340328.895483 mapread 0x00faf0 thru 0x01de47 (0xe358 bytes) 000348 1763340328.913159 trunc from 0x031aad to 0x03c13c (0xa690 bytes) 000349 1763340328.959992 MIRROR_RESYNC 3 mirrors 000351 1763340329.738497 write 0x003b26 thru 0x00859d (0x4a78 bytes) 000352 1763340329.803118 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c13d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c13d): Mirror 3: 0xbb9f1a9c (stale) Mirror 4: 0xbb9f1a9c (stale) CRC-32 checksum value for chunk [0, 0x3c13d): Mirror 1: 0x4dd6346a Mirror 5: 0x4dd6346a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c13d: skipped 000357 1763340330.853643 trunc from 0x03c13d to 0x031a7d (0xa6bf bytes) 000359 1763340330.992035 MIRROR_RESYNC 4 mirrors 000360 1763340331.939072 trunc from 0x031a7e to 0x02e0f6 (0x3987 bytes) 000361 1763340332.033834 mapread 0x02ba42 thru 0x02e0f6 (0x26b5 bytes) 000362 1763340332.055089 trunc from 0x02e0f7 to 0x000bee (0x2d508 bytes) 000364 1763340332.083571 trunc from 0x000bef to 0x00e629 (0xda3b bytes) 000368 1763340332.108580 trunc from 0x00e62a to 0x01cd79 (0xe750 bytes) 000369 1763340332.150006 trunc from 0x01cd7a to 0x01a82b (0x254e bytes) 000371 1763340332.195186 mapwrite 0x028a5d thru 0x02f2d2 (0x6876 bytes) 000374 1763340332.323799 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2f2d3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f2d3): Mirror 3: 0xa168d58f (stale) Mirror 4: 0xa168d58f (stale) Mirror 5: 0xa168d58f (stale) CRC-32 checksum value for chunk [0, 0x2f2d3): Mirror 1: 0x81c7d1df Mirror 6: 0x81c7d1df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f2d3: skipped 000375 1763340333.027789 mapwrite 0x0112d0 thru 0x014774 (0x34a5 bytes) 000378 1763340333.207201 mapread 0x02707d thru 0x02f2d2 (0x8256 bytes) 000384 1763340333.232324 mapwrite 0x03ff40 thru 0x03ffff (0xc0 bytes) 000385 1763340333.325674 mapread 0x027143 thru 0x036123 (0xefe1 bytes) 000386 1763340333.335593 mapwrite 0x025ebe thru 0x0304ea (0xa62d bytes) 000387 1763340333.379720 mapread 0x008ec5 thru 0x00dfe0 (0x511c bytes) 000389 1763340333.392744 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7cea98f4 (stale) Mirror 4: 0x7cea98f4 (stale) Mirror 5: 0x7cea98f4 (stale) Mirror 6: 0x64bd8c78 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf5a646a2 Mirror 7: 0xf5a646a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000391 1763340334.241593 trunc from 0x040000 to 0x03cebb (0x3144 bytes) 000393 1763340334.356927 trunc from 0x03cebc to 0x03b1ea (0x1cd1 bytes) 000397 1763340334.389081 MIRROR_RESYNC 6 mirrors 000399 1763340335.591561 mapread 0x02ada0 thru 0x03665a (0xb8bb bytes) 000400 1763340335.648627 trunc from 0x03b1eb to 0x023d3f (0x174ab bytes) 000401 1763340335.769116 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23d40): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23d40): Mirror 3: 0x2473ce10 (stale) Mirror 4: 0x2473ce10 (stale) Mirror 5: 0x2473ce10 (stale) Mirror 6: 0x2473ce10 (stale) Mirror 7: 0x2473ce10 (stale) CRC-32 checksum value for chunk [0, 0x23d40): Mirror 1: 0x2473ce10 Mirror 8: 0x2473ce10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23d40: skipped 000403 1763340336.795306 trunc from 0x023d40 to 0x02a175 (0x6436 bytes) 000404 1763340336.911338 write 0x01180d thru 0x019c67 (0x845b bytes) 000410 1763340336.931854 write 0x00b987 thru 0x0160ef (0xa769 bytes) 000412 1763340336.945849 trunc from 0x02a176 to 0x019e7d (0x102f8 bytes) 000413 1763340336.979233 write 0x020bc9 thru 0x0295cd (0x8a05 bytes) 000415 1763340336.982226 trunc from 0x0295ce to 0x03f4be (0x15ef1 bytes) 000416 1763340337.007989 read 0x038298 thru 0x03f4be (0x7227 bytes) 000417 1763340337.016154 mapread 0x006d3d thru 0x008637 (0x18fb bytes) 000418 1763340337.017075 write 0x03ae43 thru 0x03ffff (0x51bd bytes) 000419 1763340337.018078 mapread 0x02d942 thru 0x032074 (0x4733 bytes) 000421 1763340337.033762 trunc from 0x040000 to 0x00c220 (0x33ddf bytes) 000422 1763340337.069371 MIRROR_RESYNC 7 mirrors 000423 1763340338.219681 mapread 0x001359 thru 0x00c220 (0xaec8 bytes) 000428 1763340338.250140 read 0x005133 thru 0x009d4a (0x4c18 bytes) 000430 1763340338.250876 mapread 0x002e15 thru 0x0048ed (0x1ad9 bytes) 000435 1763340338.253972 mapwrite 0x029168 thru 0x02a2e5 (0x117e bytes) 000437 1763340338.473543 read 0x0253ee thru 0x0298f8 (0x450b bytes) 000439 1763340338.474699 read 0x01e820 thru 0x0282b5 (0x9a96 bytes) 000440 1763340338.494165 trunc from 0x02a2e6 to 0x0200db (0xa20a bytes) 000441 1763340338.515113 mapread 0x0023df thru 0x00f2a5 (0xcec7 bytes) 000442 1763340338.516256 mapread 0x012d0f thru 0x014b5c (0x1e4e bytes) 000446 1763340338.519446 write 0x0220d1 thru 0x0314e4 (0xf414 bytes) 000447 1763340338.530696 mapwrite 0x014742 thru 0x02241e (0xdcdd bytes) 000449 1763340338.589897 write 0x036f10 thru 0x03ffff (0x90f0 bytes) 000450 1763340338.591329 read 0x02f503 thru 0x031879 (0x2377 bytes) 000452 1763340338.592336 trunc from 0x040000 to 0x028900 (0x176ff bytes) 000453 1763340338.611437 trunc from 0x028901 to 0x02b812 (0x2f12 bytes) 000454 1763340338.624236 mapread 0x00a24c thru 0x00ddb6 (0x3b6b bytes) 000456 1763340338.632475 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b813): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b813): Mirror 3: 0xbf9e7c7 (stale) Mirror 4: 0xbf9e7c7 (stale) Mirror 5: 0xbf9e7c7 (stale) Mirror 6: 0xbf9e7c7 (stale) Mirror 7: 0xbf9e7c7 (stale) Mirror 8: 0xbf9e7c7 (stale) CRC-32 checksum value for chunk [0, 0x2b813): Mirror 1: 0x47e8ca2 Mirror 9: 0x47e8ca2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b813: skipped 000457 1763340339.592486 write 0x03a2e8 thru 0x03fadd (0x57f6 bytes) 000458 1763340339.662895 mapread 0x01b273 thru 0x028421 (0xd1af bytes) 000459 1763340339.699444 mapread 0x024caf thru 0x02a0f6 (0x5448 bytes) 000460 1763340339.700393 mapwrite 0x037803 thru 0x03ee5c (0x765a bytes) 000462 1763340339.761297 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3fade): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fade): Mirror 3: 0x532aa271 (stale) Mirror 4: 0x532aa271 (stale) Mirror 5: 0x532aa271 (stale) Mirror 6: 0x532aa271 (stale) Mirror 7: 0x532aa271 (stale) Mirror 8: 0x532aa271 (stale) Mirror 9: 0xf62bf52e (stale) CRC-32 checksum value for chunk [0, 0x3fade): Mirror 1: 0xc3f9575a Mirror 10: 0xc3f9575a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fade: skipped 000463 1763340340.513878 trunc from 0x03fade to 0x025844 (0x1a299 bytes) 000464 1763340340.589681 mapread 0x007a9f thru 0x0179bd (0xff1f bytes) 000465 1763340340.631003 trunc from 0x025845 to 0x02bc80 (0x643c bytes) 000467 1763340340.660576 trunc from 0x02bc81 to 0x01a257 (0x11a29 bytes) 000469 1763340340.684093 read 0x00a401 thru 0x0174de (0xd0de bytes) 000471 1763340340.685302 write 0x003980 thru 0x00b286 (0x7907 bytes) 000472 1763340340.692898 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a258): Mirror 4: 0xdcfa6af0 (stale) Mirror 5: 0xdcfa6af0 (stale) Mirror 6: 0xdcfa6af0 (stale) Mirror 7: 0xdcfa6af0 (stale) Mirror 8: 0xdcfa6af0 (stale) Mirror 9: 0xed0cbb0b (stale) Mirror 10: 0xed0cbb0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a258): Mirror 4: 0xdcfa6af0 (stale) Mirror 5: 0xdcfa6af0 (stale) Mirror 6: 0xdcfa6af0 (stale) Mirror 7: 0xdcfa6af0 (stale) Mirror 8: 0xdcfa6af0 (stale) Mirror 9: 0xed0cbb0b (stale) Mirror 10: 0xed0cbb0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a258: skipped 000474 1763340341.182936 trunc from 0x01a258 to 0x00bc5e (0xe5f9 bytes) 000476 1763340341.202662 mapread 0x002dbd thru 0x00bc5e (0x8ea2 bytes) 000477 1763340341.220929 write 0x02cb6b thru 0x032cf0 (0x6186 bytes) 000478 1763340341.222092 write 0x029842 thru 0x034885 (0xb044 bytes) 000480 1763340341.234137 mapread 0x019ab7 thru 0x01bbe7 (0x2131 bytes) 000481 1763340341.247887 mapread 0x028e78 thru 0x02e191 (0x531a bytes) 000485 1763340341.250043 mapread 0x017bb9 thru 0x026e59 (0xf2a1 bytes) 000487 1763340341.253159 write 0x03b294 thru 0x03ffff (0x4d6c bytes) 000489 1763340341.255533 mapread 0x00fd1f thru 0x019e3b (0xa11d bytes) 000491 1763340341.268253 mapread 0x008020 thru 0x014356 (0xc337 bytes) 000492 1763340341.269480 mapwrite 0x02d84c thru 0x035d26 (0x84db bytes) 000494 1763340341.364410 write 0x029086 thru 0x037552 (0xe4cd bytes) 000495 1763340341.365836 write 0x02237d thru 0x02b2c2 (0x8f46 bytes) 000496 1763340341.372436 trunc from 0x040000 to 0x00c0f1 (0x33f0e bytes) 000497 1763340341.411348 read 0x000a6c thru 0x00c0f1 (0xb686 bytes) 000501 1763340341.420092 read 0x00999b thru 0x00a4b2 (0xb18 bytes) 000502 1763340341.420733 mapwrite 0x01aa77 thru 0x028e00 (0xe38a bytes) 000503 1763340341.508639 mapwrite 0x005b1e thru 0x00e1ee (0x86d1 bytes) 000507 1763340341.614992 write 0x028a87 thru 0x02e8ca (0x5e44 bytes) 000508 1763340341.616316 trunc from 0x02e8cb to 0x02ccc1 (0x1c09 bytes) 000509 1763340341.646180 read 0x001769 thru 0x005034 (0x38cc bytes) 000510 1763340341.648109 mapwrite 0x02dbbf thru 0x03c63f (0xea81 bytes) 000512 1763340341.739428 mapread 0x01da83 thru 0x023570 (0x5aee bytes) 000515 1763340341.742168 mapwrite 0x017a14 thru 0x01de85 (0x6472 bytes) 000517 1763340341.786823 write 0x02a3b0 thru 0x02c13f (0x1d90 bytes) 000519 1763340341.788180 read 0x02ee10 thru 0x034359 (0x554a bytes) 000521 1763340341.789149 mapread 0x01694d thru 0x01a443 (0x3af7 bytes) 000526 1763340341.791819 read 0x024d60 thru 0x03029e (0xb53f bytes) 000527 1763340341.792484 write 0x014af7 thru 0x018ab4 (0x3fbe bytes) 000528 1763340341.793548 mapread 0x00c0e0 thru 0x00e5d0 (0x24f1 bytes) 000529 1763340341.794424 read 0x01c640 thru 0x022dda (0x679b bytes) 000530 1763340341.795056 write 0x0356c6 thru 0x03de1c (0x8757 bytes) 000531 1763340341.796238 trunc from 0x03de1d to 0x00b409 (0x32a13 bytes) 000532 1763340341.828910 write 0x039048 thru 0x039ad8 (0xa91 bytes) 000533 1763340341.831031 read 0x025e76 thru 0x02a31c (0x44a7 bytes) 000534 1763340341.845703 read 0x001480 thru 0x00d4ca (0xc04b bytes) 000535 1763340341.846376 trunc from 0x039ad9 to 0x004ecd (0x34c0b bytes) 000540 1763340341.892335 read 0x0041f9 thru 0x004ecd (0xcd5 bytes) 000541 1763340341.896209 trunc from 0x004ece to 0x014101 (0xf234 bytes) 000542 1763340341.929152 MIRROR_RESYNC 8 mirrors 000544 1763340343.343313 read 0x009f98 thru 0x014101 (0xa16a bytes) 000545 1763340343.362249 write 0x0054a0 thru 0x013deb (0xe94c bytes) 000546 1763340343.448733 read 0x006519 thru 0x00e0be (0x7ba6 bytes) 000548 1763340343.450304 write 0x03cde1 thru 0x03ffff (0x321f bytes) 000550 1763340343.451929 mapwrite 0x008499 thru 0x01800d (0xfb75 bytes) 000552 1763340343.539383 mapread 0x033cdd thru 0x03f6ef (0xba13 bytes) 000553 1763340343.541005 read 0x03f13e thru 0x03ffff (0xec2 bytes) 000554 1763340343.542386 mapwrite 0x021fe8 thru 0x031218 (0xf231 bytes) 000555 1763340343.587365 read 0x025bab thru 0x02626e (0x6c4 bytes) 000556 1763340343.588072 mapwrite 0x016b92 thru 0x01c319 (0x5788 bytes) 000557 1763340343.647293 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x84d0851e (stale) Mirror 5: 0x84d0851e (stale) Mirror 6: 0x84d0851e (stale) Mirror 7: 0x84d0851e (stale) Mirror 8: 0x84d0851e (stale) Mirror 9: 0x84d0851e (stale) Mirror 10: 0x84d0851e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa6ca52a7 Mirror 11: 0xa6ca52a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000558 1763340344.650449 mapread 0x02cd57 thru 0x034474 (0x771e bytes) 000560 1763340344.690730 write 0x038a00 thru 0x039229 (0x82a bytes) 000561 1763340344.778145 write 0x0026ff thru 0x00ba04 (0x9306 bytes) 000562 1763340344.790577 trunc from 0x040000 to 0x00f74a (0x308b5 bytes) 000563 1763340344.825543 mapread 0x0061e6 thru 0x00f74a (0x9565 bytes) 000565 1763340344.827113 trunc from 0x00f74b to 0x01a9b2 (0xb268 bytes) 000569 1763340344.852009 trunc from 0x01a9b3 to 0x002823 (0x1818f bytes) 000572 1763340344.874030 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2824): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2824): Mirror 4: 0xda4723a2 (stale) Mirror 5: 0xda4723a2 (stale) Mirror 6: 0xda4723a2 (stale) Mirror 7: 0xda4723a2 (stale) Mirror 8: 0xda4723a2 (stale) Mirror 9: 0xda4723a2 (stale) Mirror 10: 0xda4723a2 (stale) Mirror 11: 0xda4723a2 (stale) CRC-32 checksum value for chunk [0, 0x2824): Mirror 1: 0x6fd88987 Mirror 12: 0x6fd88987 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2824: skipped 000573 1763340345.668916 write 0x007322 thru 0x00cf59 (0x5c38 bytes) 000574 1763340345.760955 trunc from 0x00cf5a to 0x01ead3 (0x11b7a bytes) 000576 1763340345.782434 trunc from 0x01ead4 to 0x037632 (0x18b5f bytes) 000582 1763340345.802074 mapwrite 0x01eb1e thru 0x02e474 (0xf957 bytes) 000583 1763340345.947197 mapwrite 0x03a05b thru 0x03ffff (0x5fa5 bytes) 000584 1763340346.028402 write 0x01ae85 thru 0x029063 (0xe1df bytes) 000586 1763340346.034821 read 0x0303e3 thru 0x03eb39 (0xe757 bytes) 000587 1763340346.044247 write 0x03ff0f thru 0x03ffff (0xf1 bytes) 000589 1763340346.050665 write 0x03a408 thru 0x03ffff (0x5bf8 bytes) 000590 1763340346.051751 trunc from 0x040000 to 0x009542 (0x36abd bytes) 000591 1763340346.079867 write 0x00e383 thru 0x018fe9 (0xac67 bytes) 000596 1763340346.085309 write 0x03e851 thru 0x03f77b (0xf2b bytes) 000597 1763340346.086272 mapread 0x006824 thru 0x01358d (0xcd6a bytes) 000598 1763340346.107600 MIRROR_RESYNC 10 mirrors 000599 1763340347.649093 trunc from 0x03f77c to 0x025504 (0x1a277 bytes) 000601 1763340347.730940 mapwrite 0x02a4d9 thru 0x02e376 (0x3e9e bytes) 000602 1763340347.837821 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2e377): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e377): Mirror 4: 0xaee46e3a (stale) Mirror 5: 0xaee46e3a (stale) Mirror 6: 0xaee46e3a (stale) Mirror 7: 0xaee46e3a (stale) Mirror 8: 0xaee46e3a (stale) Mirror 9: 0xaee46e3a (stale) Mirror 10: 0xaee46e3a (stale) Mirror 11: 0xaee46e3a (stale) Mirror 12: 0xaee46e3a (stale) CRC-32 checksum value for chunk [0, 0x2e377): Mirror 1: 0xb0fe9fd7 Mirror 13: 0xb0fe9fd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e377: skipped 000603 1763340348.899296 trunc from 0x02e377 to 0x03a7f8 (0xc482 bytes) 000604 1763340348.988892 write 0x021aeb thru 0x03022c (0xe742 bytes) 000605 1763340349.008071 mapread 0x00f5cb thru 0x00f655 (0x8b bytes) 000606 1763340349.019157 mapread 0x011f75 thru 0x01ad4a (0x8dd6 bytes) 000609 1763340349.035967 mapwrite 0x030fa7 thru 0x0392b1 (0x830b bytes) 000611 1763340349.096033 trunc from 0x03a7f9 to 0x0065d5 (0x34223 bytes) 000612 1763340349.119018 read 0x001ac0 thru 0x0065d5 (0x4b16 bytes) 000613 1763340349.129174 mapread 0x00374f thru 0x0065d5 (0x2e87 bytes) 000617 1763340349.131476 mapwrite 0x020346 thru 0x026aa5 (0x6760 bytes) 000618 1763340349.213228 mapread 0x022b2d thru 0x026aa5 (0x3f79 bytes) 000619 1763340349.214395 mapread 0x007055 thru 0x013d73 (0xcd1f bytes) 000620 1763340349.235320 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26aa6): Mirror 4: 0xcb26f1d5 (stale) Mirror 5: 0xcb26f1d5 (stale) Mirror 6: 0xcb26f1d5 (stale) Mirror 7: 0xcb26f1d5 (stale) Mirror 8: 0xcb26f1d5 (stale) Mirror 10: 0xcb26f1d5 (stale) Mirror 11: 0xcb26f1d5 (stale) Mirror 12: 0xcb26f1d5 (stale) Mirror 13: 0xcb26f1d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26aa6): Mirror 4: 0xcb26f1d5 (stale) Mirror 5: 0xcb26f1d5 (stale) Mirror 6: 0xcb26f1d5 (stale) Mirror 7: 0xcb26f1d5 (stale) Mirror 8: 0xcb26f1d5 (stale) Mirror 10: 0xcb26f1d5 (stale) Mirror 11: 0xcb26f1d5 (stale) Mirror 12: 0xcb26f1d5 (stale) Mirror 13: 0xcb26f1d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26aa6: skipped 000622 1763340349.911701 mapread 0x017669 thru 0x024ac7 (0xd45f bytes) 000623 1763340349.942922 trunc from 0x026aa6 to 0x01d71d (0x9388 bytes) 000624 1763340349.965834 trunc from 0x01d71e to 0x01dea6 (0x789 bytes) 000625 1763340349.990473 read 0x005e06 thru 0x00c375 (0x6570 bytes) 000626 1763340350.000338 trunc from 0x01dea7 to 0x017e60 (0x6046 bytes) 000627 1763340350.020888 write 0x036fe9 thru 0x03ffff (0x9017 bytes) 000629 1763340350.022749 mapwrite 0x039fe5 thru 0x03ffff (0x601b bytes) 000631 1763340350.063742 mapwrite 0x0251c8 thru 0x0300a0 (0xaed9 bytes) 000633 1763340350.140683 write 0x02070f thru 0x0282cf (0x7bc1 bytes) 000637 1763340350.144055 read 0x012839 thru 0x021181 (0xe949 bytes) 000639 1763340350.156834 trunc from 0x040000 to 0x003ca1 (0x3c35e bytes) 000641 1763340350.187866 write 0x01f537 thru 0x02812c (0x8bf6 bytes) 000642 1763340350.189352 trunc from 0x02812d to 0x03b34e (0x13222 bytes) 000643 1763340350.210613 read 0x031e60 thru 0x039dc7 (0x7f68 bytes) 000644 1763340350.228624 read 0x01a7e8 thru 0x0291ae (0xe9c7 bytes) 000646 1763340350.230308 read 0x00e388 thru 0x01dcdd (0xf956 bytes) 000648 1763340350.231798 write 0x00671a thru 0x00815d (0x1a44 bytes) 000653 1763340350.235671 trunc from 0x03b34f to 0x014ba6 (0x267a8 bytes) 000655 1763340350.261578 mapwrite 0x019b09 thru 0x0288c1 (0xedb9 bytes) 000656 1763340350.335759 trunc from 0x0288c2 to 0x00b82a (0x1d097 bytes) 000659 1763340350.358102 mapread 0x002cc5 thru 0x009d0b (0x7047 bytes) 000660 1763340350.359152 write 0x01dccb thru 0x025f0a (0x8240 bytes) 000662 1763340350.378822 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25f0b): Mirror 5: 0x36a28ba5 (stale) Mirror 6: 0x36a28ba5 (stale) Mirror 7: 0x36a28ba5 (stale) Mirror 8: 0x36a28ba5 (stale) Mirror 10: 0x36a28ba5 (stale) Mirror 11: 0x36a28ba5 (stale) Mirror 12: 0x36a28ba5 (stale) Mirror 13: 0x36a28ba5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25f0b): Mirror 5: 0x36a28ba5 (stale) Mirror 6: 0x36a28ba5 (stale) Mirror 7: 0x36a28ba5 (stale) Mirror 8: 0x36a28ba5 (stale) Mirror 10: 0x36a28ba5 (stale) Mirror 11: 0x36a28ba5 (stale) Mirror 12: 0x36a28ba5 (stale) Mirror 13: 0x36a28ba5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25f0b: skipped 000663 1763340350.951296 mapwrite 0x032bfa thru 0x03ffff (0xd406 bytes) 000664 1763340351.050763 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x135e275d (stale) Mirror 6: 0x135e275d (stale) Mirror 7: 0x135e275d (stale) Mirror 8: 0x135e275d (stale) Mirror 10: 0x135e275d (stale) Mirror 11: 0x135e275d (stale) Mirror 12: 0x135e275d (stale) Mirror 13: 0x829c6371 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbfc0a3a Mirror 14: 0xbfc0a3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000666 1763340352.052444 trunc from 0x040000 to 0x03bef5 (0x410a bytes) 000668 1763340352.166375 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3bef6): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bef6): Mirror 5: 0x3c86205a (stale) Mirror 6: 0x3c86205a (stale) Mirror 7: 0x3c86205a (stale) Mirror 8: 0x3c86205a (stale) Mirror 10: 0x3c86205a (stale) Mirror 11: 0x3c86205a (stale) Mirror 12: 0x3c86205a (stale) Mirror 13: 0x4cab0b32 (stale) Mirror 14: 0x7b986bfd (stale) CRC-32 checksum value for chunk [0, 0x3bef6): Mirror 1: 0x7b986bfd Mirror 15: 0x7b986bfd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bef6: skipped 000669 1763340353.188333 write 0x01b632 thru 0x0293f7 (0xddc6 bytes) 000675 1763340353.291376 mapwrite 0x03220e thru 0x03ffff (0xddf2 bytes) 000676 1763340353.383144 trunc from 0x040000 to 0x02e97a (0x11685 bytes) 000677 1763340353.412298 write 0x030531 thru 0x03ffff (0xfacf bytes) 000678 1763340353.414089 trunc from 0x040000 to 0x0327b6 (0xd849 bytes) 000680 1763340353.436040 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x327b7): Mirror 6: 0x89f17b60 (stale) Mirror 7: 0x89f17b60 (stale) Mirror 8: 0x89f17b60 (stale) Mirror 10: 0x89f17b60 (stale) Mirror 11: 0x89f17b60 (stale) Mirror 12: 0x89f17b60 (stale) Mirror 13: 0xcc63fb34 (stale) Mirror 14: 0x5b79188e (stale) Mirror 15: 0x5b79188e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x327b7): Mirror 6: 0x89f17b60 (stale) Mirror 7: 0x89f17b60 (stale) Mirror 8: 0x89f17b60 (stale) Mirror 10: 0x89f17b60 (stale) Mirror 11: 0x89f17b60 (stale) Mirror 12: 0x89f17b60 (stale) Mirror 13: 0xcc63fb34 (stale) Mirror 14: 0x5b79188e (stale) Mirror 15: 0x5b79188e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x327b7: skipped 000681 1763340354.122568 read 0x013ba5 thru 0x022a78 (0xeed4 bytes) 000684 1763340354.142899 mapwrite 0x03cba1 thru 0x03ffff (0x345f bytes) 000686 1763340354.227042 trunc from 0x040000 to 0x03030e (0xfcf1 bytes) 000688 1763340354.247873 trunc from 0x03030f to 0x02a6d6 (0x5c38 bytes) 000689 1763340354.273477 mapread 0x00a649 thru 0x00c456 (0x1e0e bytes) 000691 1763340354.288105 MIRROR_RESYNC 10 mirrors 000692 1763340355.891657 mapread 0x028f7d thru 0x02a6d6 (0x175a bytes) 000693 1763340355.908146 read 0x0116b2 thru 0x01fc01 (0xe550 bytes) 000695 1763340355.917082 trunc from 0x02a6d7 to 0x010712 (0x19fc4 bytes) 000696 1763340356.000265 mapwrite 0x0124c9 thru 0x0207e4 (0xe31c bytes) 000697 1763340356.110596 mapread 0x00ed21 thru 0x00f0e8 (0x3c8 bytes) 000699 1763340356.113464 mapwrite 0x02ab9d thru 0x02ae76 (0x2da bytes) 000700 1763340356.176366 mapread 0x0057f6 thru 0x00b32e (0x5b39 bytes) 000701 1763340356.186088 trunc from 0x02ae77 to 0x02ff16 (0x50a0 bytes) 000703 1763340356.206170 read 0x0071d9 thru 0x008d26 (0x1b4e bytes) 000704 1763340356.206836 write 0x02f73e thru 0x03441d (0x4ce0 bytes) 000705 1763340356.218003 mapread 0x008853 thru 0x008ef9 (0x6a7 bytes) 000707 1763340356.222724 MIRROR_RESYNC 10 mirrors 000708 1763340357.551478 read 0x021ca9 thru 0x028809 (0x6b61 bytes) 000711 1763340357.562572 write 0x034180 thru 0x03bcd4 (0x7b55 bytes) 000714 1763340357.653888 mapread 0x00b9bc thru 0x019c06 (0xe24b bytes) 000717 1763340357.691364 mapwrite 0x0216f8 thru 0x0249e5 (0x32ee bytes) 000720 1763340357.744472 write 0x0271fb thru 0x02f02f (0x7e35 bytes) 000722 1763340357.750961 write 0x036108 thru 0x03899c (0x2895 bytes) 000726 1763340357.753716 trunc from 0x03bcd5 to 0x0287ea (0x134ea bytes) 000728 1763340357.777986 mapwrite 0x010316 thru 0x012c85 (0x2970 bytes) 000729 1763340357.826380 trunc from 0x0287eb to 0x003274 (0x25576 bytes) 000730 1763340357.846814 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3275): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3275): Mirror 6: 0xce046958 (stale) Mirror 7: 0xce046958 (stale) Mirror 8: 0xce046958 (stale) Mirror 10: 0xce046958 (stale) Mirror 11: 0xce046958 (stale) Mirror 12: 0xce046958 (stale) Mirror 13: 0xce046958 (stale) Mirror 14: 0xce046958 (stale) Mirror 15: 0xce046958 (stale) CRC-32 checksum value for chunk [0, 0x3275): Mirror 1: 0xce046958 Mirror 16: 0xce046958 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3275: skipped 000731 1763340358.758805 trunc from 0x003275 to 0x0160f8 (0x12e84 bytes) 000733 1763340358.894754 trunc from 0x0160f9 to 0x02b4d2 (0x153da bytes) 000734 1763340358.935578 read 0x00a670 thru 0x0124c3 (0x7e54 bytes) 000735 1763340358.946773 write 0x0121e9 thru 0x0173d1 (0x51e9 bytes) 000739 1763340358.967260 write 0x0063aa thru 0x00c3d5 (0x602c bytes) 000740 1763340358.985975 MIRROR_RESYNC 11 mirrors 000741 1763340360.839015 trunc from 0x02b4d3 to 0x03c2a5 (0x10dd3 bytes) 000742 1763340360.905859 read 0x002d75 thru 0x01062a (0xd8b6 bytes) 000743 1763340360.923882 mapread 0x00ed2e thru 0x01d6b3 (0xe986 bytes) 000744 1763340360.969596 mapread 0x014467 thru 0x01564a (0x11e4 bytes) 000745 1763340360.970565 mapwrite 0x002667 thru 0x007a4b (0x53e5 bytes) 000746 1763340361.025305 mapread 0x009805 thru 0x00d58f (0x3d8b bytes) 000747 1763340361.031903 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3c2a6): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2a6): Mirror 6: 0x71b803aa (stale) Mirror 7: 0x71b803aa (stale) Mirror 8: 0x71b803aa (stale) Mirror 10: 0x71b803aa (stale) Mirror 11: 0x71b803aa (stale) Mirror 12: 0x71b803aa (stale) Mirror 13: 0x71b803aa (stale) Mirror 14: 0x71b803aa (stale) Mirror 15: 0x71b803aa (stale) Mirror 16: 0x71b803aa (stale) CRC-32 checksum value for chunk [0, 0x3c2a6): Mirror 1: 0x4cb02a0d Mirror 17: 0x4cb02a0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2a6: skipped 000750 1763340362.201161 mapread 0x03561f thru 0x038481 (0x2e63 bytes) 000752 1763340362.254254 write 0x028e3c thru 0x02adcf (0x1f94 bytes) 000753 1763340362.373015 mapread 0x026541 thru 0x027da0 (0x1860 bytes) 000755 1763340362.374414 read 0x01a7cb thru 0x026cd7 (0xc50d bytes) 000760 1763340362.403468 write 0x0181eb thru 0x0261b1 (0xdfc7 bytes) 000762 1763340362.412024 mapwrite 0x00f043 thru 0x018d58 (0x9d16 bytes) 000767 1763340362.474739 read 0x031a53 thru 0x03c2a5 (0xa853 bytes) 000772 1763340362.508376 trunc from 0x03c2a6 to 0x01c2a7 (0x1fffe bytes) 000775 1763340362.550993 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1c2a8): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c2a8): Mirror 6: 0xb513d62e (stale) Mirror 7: 0xb513d62e (stale) Mirror 8: 0xb513d62e (stale) Mirror 10: 0xb513d62e (stale) Mirror 11: 0xb513d62e (stale) Mirror 12: 0xb513d62e (stale) Mirror 13: 0xb513d62e (stale) Mirror 14: 0xb513d62e (stale) Mirror 15: 0xb513d62e (stale) Mirror 16: 0xb513d62e (stale) Mirror 17: 0xeb2e036b (stale) CRC-32 checksum value for chunk [0, 0x1c2a8): Mirror 1: 0x538cbce6 Mirror 18: 0x538cbce6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c2a8: skipped 000776 1763340363.532083 trunc from 0x01c2a8 to 0x005f8c (0x1631b bytes) 000778 1763340363.660687 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5f8d): Mirror 6: 0x352324f6 (stale) Mirror 7: 0x352324f6 (stale) Mirror 10: 0x352324f6 (stale) Mirror 11: 0x352324f6 (stale) Mirror 12: 0x352324f6 (stale) Mirror 13: 0x352324f6 (stale) Mirror 14: 0x352324f6 (stale) Mirror 15: 0x352324f6 (stale) Mirror 16: 0x352324f6 (stale) Mirror 17: 0xb45283f5 (stale) Mirror 18: 0xb45283f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5f8d): Mirror 6: 0x352324f6 (stale) Mirror 7: 0x352324f6 (stale) Mirror 10: 0x352324f6 (stale) Mirror 11: 0x352324f6 (stale) Mirror 12: 0x352324f6 (stale) Mirror 13: 0x352324f6 (stale) Mirror 14: 0x352324f6 (stale) Mirror 15: 0x352324f6 (stale) Mirror 16: 0x352324f6 (stale) Mirror 17: 0xb45283f5 (stale) Mirror 18: 0xb45283f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5f8d: skipped 000779 1763340364.178593 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x5f8d): Mirror 6: 0x352324f6 (stale) Mirror 7: 0x352324f6 (stale) Mirror 10: 0x352324f6 (stale) Mirror 11: 0x352324f6 (stale) Mirror 12: 0x352324f6 (stale) Mirror 13: 0x352324f6 (stale) Mirror 14: 0x352324f6 (stale) Mirror 15: 0x352324f6 (stale) Mirror 17: 0xb45283f5 (stale) Mirror 18: 0xb45283f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5f8d): Mirror 6: 0x352324f6 (stale) Mirror 7: 0x352324f6 (stale) Mirror 10: 0x352324f6 (stale) Mirror 11: 0x352324f6 (stale) Mirror 12: 0x352324f6 (stale) Mirror 13: 0x352324f6 (stale) Mirror 14: 0x352324f6 (stale) Mirror 15: 0x352324f6 (stale) Mirror 17: 0xb45283f5 (stale) Mirror 18: 0xb45283f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5f8d: skipped 000780 1763340364.697817 write 0x027379 thru 0x02d1cc (0x5e54 bytes) 000781 1763340364.699283 read 0x011449 thru 0x01bdc5 (0xa97d bytes) 000782 1763340364.712665 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d1cd): Mirror 6: 0xef32956e (stale) Mirror 7: 0xef32956e (stale) Mirror 11: 0xef32956e (stale) Mirror 12: 0xef32956e (stale) Mirror 13: 0xef32956e (stale) Mirror 14: 0xef32956e (stale) Mirror 15: 0xef32956e (stale) Mirror 17: 0x4be4c030 (stale) Mirror 18: 0x82047bb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d1cd): Mirror 6: 0xef32956e (stale) Mirror 7: 0xef32956e (stale) Mirror 11: 0xef32956e (stale) Mirror 12: 0xef32956e (stale) Mirror 13: 0xef32956e (stale) Mirror 14: 0xef32956e (stale) Mirror 15: 0xef32956e (stale) Mirror 17: 0x4be4c030 (stale) Mirror 18: 0x82047bb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d1cd: skipped 000784 1763340365.293052 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d1cd): Mirror 7: 0xef32956e (stale) Mirror 11: 0xef32956e (stale) Mirror 12: 0xef32956e (stale) Mirror 13: 0xef32956e (stale) Mirror 14: 0xef32956e (stale) Mirror 15: 0xef32956e (stale) Mirror 17: 0x4be4c030 (stale) Mirror 18: 0x82047bb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d1cd): Mirror 7: 0xef32956e (stale) Mirror 11: 0xef32956e (stale) Mirror 12: 0xef32956e (stale) Mirror 13: 0xef32956e (stale) Mirror 14: 0xef32956e (stale) Mirror 15: 0xef32956e (stale) Mirror 17: 0x4be4c030 (stale) Mirror 18: 0x82047bb0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d1cd: skipped 000785 1763340365.857877 trunc from 0x02d1cd to 0x034c30 (0x7a64 bytes) 000786 1763340365.885732 write 0x0099b3 thru 0x00bb04 (0x2152 bytes) 000787 1763340365.896422 trunc from 0x034c31 to 0x0356b8 (0xa88 bytes) 000788 1763340365.910596 mapwrite 0x00fec6 thru 0x01e1c7 (0xe302 bytes) 000790 1763340366.026105 write 0x03008d thru 0x03da7d (0xd9f1 bytes) 000791 1763340366.042935 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3da7e): Mirror 11: 0x288554cd (stale) Mirror 12: 0x288554cd (stale) Mirror 13: 0x288554cd (stale) Mirror 14: 0x288554cd (stale) Mirror 15: 0x288554cd (stale) Mirror 17: 0xd14732c6 (stale) Mirror 18: 0x45798981 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da7e): Mirror 11: 0x288554cd (stale) Mirror 12: 0x288554cd (stale) Mirror 13: 0x288554cd (stale) Mirror 14: 0x288554cd (stale) Mirror 15: 0x288554cd (stale) Mirror 17: 0xd14732c6 (stale) Mirror 18: 0x45798981 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da7e: skipped 000792 1763340366.603711 read 0x009725 thru 0x00e69b (0x4f77 bytes) 000793 1763340366.614083 read 0x002613 thru 0x0099fe (0x73ec bytes) 000794 1763340366.630272 mapwrite 0x023d6b thru 0x02ad22 (0x6fb8 bytes) 000795 1763340366.683164 write 0x03efed thru 0x03ffff (0x1013 bytes) 000800 1763340366.686077 read 0x01609d thru 0x01e85e (0x87c2 bytes) 000801 1763340366.686812 mapwrite 0x0000dc thru 0x0046ce (0x45f3 bytes) 000803 1763340366.785493 MIRROR_RESYNC 8 mirrors 000804 1763340368.119872 mapread 0x019ab2 thru 0x020afa (0x7049 bytes) 000805 1763340368.148904 trunc from 0x040000 to 0x0175d8 (0x28a27 bytes) 000807 1763340368.268753 read 0x013ba7 thru 0x0175d8 (0x3a32 bytes) 000808 1763340368.282060 mapread 0x0104d8 thru 0x015b95 (0x56be bytes) 000809 1763340368.295820 mapread 0x00423d thru 0x00d89e (0x9662 bytes) 000811 1763340368.306630 mapwrite 0x03ab8b thru 0x03ffff (0x5475 bytes) 000812 1763340368.383854 mapread 0x03bf24 thru 0x03ffff (0x40dc bytes) 000815 1763340368.392685 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x2bd4ea45 (stale) Mirror 12: 0x2bd4ea45 (stale) Mirror 13: 0x2bd4ea45 (stale) Mirror 14: 0x2bd4ea45 (stale) Mirror 15: 0x2bd4ea45 (stale) Mirror 18: 0x2bd4ea45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x2bd4ea45 (stale) Mirror 12: 0x2bd4ea45 (stale) Mirror 13: 0x2bd4ea45 (stale) Mirror 14: 0x2bd4ea45 (stale) Mirror 15: 0x2bd4ea45 (stale) Mirror 18: 0x2bd4ea45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000817 1763340368.842901 write 0x03f775 thru 0x03ffff (0x88b bytes) 000818 1763340368.852497 trunc from 0x040000 to 0x015393 (0x2ac6c bytes) 000819 1763340368.882917 mapread 0x001a3a thru 0x0053c7 (0x398e bytes) 000820 1763340368.909264 mapwrite 0x03f002 thru 0x03ffff (0xffe bytes) 000821 1763340369.014754 read 0x0376d7 thru 0x03da9e (0x63c8 bytes) 000825 1763340369.028910 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x2bd4ea45 (stale) Mirror 12: 0x2bd4ea45 (stale) Mirror 13: 0x2bd4ea45 (stale) Mirror 14: 0x2bd4ea45 (stale) Mirror 15: 0x2bd4ea45 (stale) Mirror 18: 0x2bd4ea45 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x56c418b2 Mirror 19: 0x56c418b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000826 1763340370.217905 read 0x0270cc thru 0x02a3fa (0x332f bytes) 000827 1763340370.253046 mapwrite 0x017b2c thru 0x01b13b (0x3610 bytes) 000828 1763340370.499710 write 0x001b05 thru 0x004afa (0x2ff6 bytes) 000829 1763340370.511098 mapwrite 0x03af8e thru 0x03ffff (0x5072 bytes) 000830 1763340370.557057 trunc from 0x040000 to 0x01330e (0x2ccf1 bytes) 000832 1763340370.582815 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1330f): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1330f): Mirror 11: 0x3c02e0d7 (stale) Mirror 12: 0x3c02e0d7 (stale) Mirror 13: 0x3c02e0d7 (stale) Mirror 14: 0x3c02e0d7 (stale) Mirror 15: 0x3c02e0d7 (stale) Mirror 18: 0x3c02e0d7 (stale) Mirror 19: 0x3c02e0d7 (stale) CRC-32 checksum value for chunk [0, 0x1330f): Mirror 1: 0xe5057118 Mirror 20: 0xe5057118 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1330f: skipped 000834 1763340371.816957 trunc from 0x01330f to 0x02d3fc (0x1a0ee bytes) 000837 1763340371.944269 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d3fd): Mirror 12: 0xf8fcf37a (stale) Mirror 13: 0xf8fcf37a (stale) Mirror 14: 0xf8fcf37a (stale) Mirror 15: 0xf8fcf37a (stale) Mirror 18: 0xf8fcf37a (stale) Mirror 19: 0xa83a38b9 (stale) Mirror 20: 0x7be94be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d3fd): Mirror 12: 0xf8fcf37a (stale) Mirror 13: 0xf8fcf37a (stale) Mirror 14: 0xf8fcf37a (stale) Mirror 15: 0xf8fcf37a (stale) Mirror 18: 0xf8fcf37a (stale) Mirror 19: 0xa83a38b9 (stale) Mirror 20: 0x7be94be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d3fd: skipped 000838 1763340372.516244 trunc from 0x02d3fd to 0x0130db (0x1a321 bytes) 000841 1763340372.540784 write 0x029ed1 thru 0x02a302 (0x432 bytes) 000842 1763340372.542593 trunc from 0x02a303 to 0x020b70 (0x9792 bytes) 000843 1763340372.579306 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x20b71): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20b71): Mirror 12: 0xe7da6b95 (stale) Mirror 13: 0xe7da6b95 (stale) Mirror 14: 0xe7da6b95 (stale) Mirror 15: 0xe7da6b95 (stale) Mirror 18: 0xe7da6b95 (stale) Mirror 19: 0x7e997836 (stale) Mirror 20: 0xe857abc0 (stale) CRC-32 checksum value for chunk [0, 0x20b71): Mirror 1: 0x451528b1 Mirror 21: 0x451528b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20b71: skipped 000845 1763340373.826861 mapread 0x01173c thru 0x0121b7 (0xa7c bytes) 000846 1763340373.855342 trunc from 0x020b71 to 0x033bf1 (0x13081 bytes) 000847 1763340373.953327 MIRROR_RESYNC 9 mirrors 000848 1763340375.293287 MIRROR_RESYNC 9 mirrors 000849 1763340375.781188 write 0x03b307 thru 0x03e41e (0x3118 bytes) 000850 1763340375.855228 trunc from 0x03e41f to 0x002306 (0x3c118 bytes) 000853 1763340375.883311 trunc from 0x002307 to 0x03e379 (0x3c073 bytes) 000854 1763340375.904624 mapwrite 0x03dd91 thru 0x03ffff (0x226f bytes) 000855 1763340375.991535 read 0x021e0c thru 0x022734 (0x929 bytes) 000856 1763340376.001538 trunc from 0x040000 to 0x0188ec (0x27713 bytes) 000857 1763340376.023376 mapwrite 0x03daa7 thru 0x03ffff (0x2559 bytes) 000858 1763340376.121827 read 0x03b8a0 thru 0x03ffff (0x4760 bytes) 000861 1763340376.123458 mapread 0x01ac71 thru 0x02406c (0x93fc bytes) 000862 1763340376.137370 mapread 0x012d84 thru 0x01c59c (0x9819 bytes) 000865 1763340376.139116 mapread 0x02d9e1 thru 0x033118 (0x5738 bytes) 000868 1763340376.143694 mapwrite 0x00eb91 thru 0x01ab0c (0xbf7c bytes) 000869 1763340376.190011 write 0x02d716 thru 0x02ebdd (0x14c8 bytes) 000873 1763340376.193323 mapread 0x029671 thru 0x0379b7 (0xe347 bytes) 000874 1763340376.195114 write 0x0322ef thru 0x032795 (0x4a7 bytes) 000876 1763340376.196737 read 0x02a94e thru 0x02d32c (0x29df bytes) 000877 1763340376.197986 mapread 0x03926a thru 0x03ffff (0x6d96 bytes) 000878 1763340376.199684 read 0x03afc7 thru 0x03dd6c (0x2da6 bytes) 000879 1763340376.200187 mapread 0x02ebe0 thru 0x030055 (0x1476 bytes) 000880 1763340376.200926 trunc from 0x040000 to 0x01770e (0x288f1 bytes) 000881 1763340376.223057 mapwrite 0x0134dd thru 0x01eff1 (0xbb15 bytes) 000882 1763340376.326576 mapwrite 0x00569c thru 0x00d6ae (0x8013 bytes) 000884 1763340376.379058 read 0x00b983 thru 0x01a0e6 (0xe764 bytes) 000886 1763340376.380093 mapread 0x0036e2 thru 0x011558 (0xde77 bytes) 000887 1763340376.384949 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1eff2): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eff2): Mirror 12: 0x200f94fc (stale) Mirror 13: 0x200f94fc (stale) Mirror 14: 0x200f94fc (stale) Mirror 15: 0x200f94fc (stale) Mirror 18: 0x200f94fc (stale) Mirror 19: 0x200f94fc (stale) Mirror 20: 0x200f94fc (stale) Mirror 21: 0x200f94fc (stale) CRC-32 checksum value for chunk [0, 0x1eff2): Mirror 1: 0xbb79c42c Mirror 22: 0xbb79c42c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eff2: skipped 000889 1763340377.445717 mapwrite 0x02e58c thru 0x036aa5 (0x851a bytes) 000890 1763340377.620284 trunc from 0x036aa6 to 0x03195d (0x5148 bytes) 000891 1763340377.657974 mapwrite 0x015bd8 thru 0x0212a9 (0xb6d2 bytes) 000893 1763340377.754732 mapread 0x0311a9 thru 0x03195d (0x7b5 bytes) 000895 1763340377.758831 read 0x018e8b thru 0x01ae6b (0x1fe1 bytes) 000896 1763340377.759454 mapread 0x023ced thru 0x027fa3 (0x42b7 bytes) 000897 1763340377.770992 read 0x02d78e thru 0x03195d (0x41d0 bytes) 000898 1763340377.773743 mapwrite 0x0283f4 thru 0x0348e7 (0xc4f4 bytes) 000899 1763340377.852631 write 0x013f3f thru 0x01d0cb (0x918d bytes) 000901 1763340377.854278 read 0x00a2fb thru 0x00b7b3 (0x14b9 bytes) 000902 1763340377.854835 trunc from 0x0348e8 to 0x00ee70 (0x25a77 bytes) 000903 1763340377.889932 MIRROR_RESYNC 10 mirrors 000905 1763340379.363291 trunc from 0x00ee71 to 0x0253a7 (0x16537 bytes) 000906 1763340379.442725 trunc from 0x0253a8 to 0x03003b (0xac94 bytes) 000907 1763340379.475205 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3003c): Mirror 12: 0xc4ac2ba (stale) Mirror 13: 0xc4ac2ba (stale) Mirror 14: 0xc4ac2ba (stale) Mirror 15: 0xc4ac2ba (stale) Mirror 18: 0xc4ac2ba (stale) Mirror 20: 0xc4ac2ba (stale) Mirror 21: 0xc4ac2ba (stale) Mirror 22: 0xc4ac2ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3003c): Mirror 12: 0xc4ac2ba (stale) Mirror 13: 0xc4ac2ba (stale) Mirror 14: 0xc4ac2ba (stale) Mirror 15: 0xc4ac2ba (stale) Mirror 18: 0xc4ac2ba (stale) Mirror 20: 0xc4ac2ba (stale) Mirror 21: 0xc4ac2ba (stale) Mirror 22: 0xc4ac2ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3003c: skipped 000909 1763340379.961212 mapwrite 0x007c88 thru 0x00a60e (0x2987 bytes) 000911 1763340380.040042 mapread 0x00d829 thru 0x00dc28 (0x400 bytes) 000912 1763340380.040970 write 0x01d28b thru 0x02b3ca (0xe140 bytes) 000913 1763340380.058248 mapwrite 0x00b045 thru 0x00cf19 (0x1ed5 bytes) 000914 1763340380.129147 write 0x0309c2 thru 0x03fe42 (0xf481 bytes) 000915 1763340380.144774 write 0x00410c thru 0x00a8e2 (0x67d7 bytes) 000916 1763340380.145900 read 0x0100a1 thru 0x0193df (0x933f bytes) 000917 1763340380.151366 read 0x00135c thru 0x002708 (0x13ad bytes) 000919 1763340380.165028 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe43): Mirror 13: 0xe801ac86 (stale) Mirror 14: 0xe801ac86 (stale) Mirror 15: 0xe801ac86 (stale) Mirror 18: 0xe801ac86 (stale) Mirror 20: 0xe801ac86 (stale) Mirror 21: 0xe801ac86 (stale) Mirror 22: 0xe801ac86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe43): Mirror 13: 0xe801ac86 (stale) Mirror 14: 0xe801ac86 (stale) Mirror 15: 0xe801ac86 (stale) Mirror 18: 0xe801ac86 (stale) Mirror 20: 0xe801ac86 (stale) Mirror 21: 0xe801ac86 (stale) Mirror 22: 0xe801ac86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe43: skipped 000921 1763340380.647952 write 0x02315e thru 0x02e95c (0xb7ff bytes) 000922 1763340380.664526 mapwrite 0x005ba9 thru 0x01268b (0xcae3 bytes) 000923 1763340380.750435 mapread 0x03c79b thru 0x03fe42 (0x36a8 bytes) 000926 1763340380.764341 MIRROR_RESYNC 8 mirrors 000927 1763340382.209063 mapwrite 0x029dc8 thru 0x02a5d4 (0x80d bytes) 000928 1763340382.396665 mapread 0x00476a thru 0x013ca7 (0xf53e bytes) 000929 1763340382.420165 MIRROR_RESYNC 8 mirrors 000931 1763340383.564322 read 0x00ac0b thru 0x013de5 (0x91db bytes) 000933 1763340383.574765 read 0x02f05b thru 0x034543 (0x54e9 bytes) 000935 1763340383.592271 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) 000936 1763340383.985444 mapread 0x01ae9e thru 0x01cb80 (0x1ce3 bytes) 000938 1763340384.007646 read 0x017bff thru 0x01c537 (0x4939 bytes) 000940 1763340384.008733 read 0x00d94d thru 0x017021 (0x96d5 bytes) 000943 1763340384.034242 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) 000944 1763340384.451924 mapwrite 0x00bf84 thru 0x01b20e (0xf28b bytes) 000945 1763340384.661018 trunc from 0x03fe43 to 0x03e87f (0x15c3 bytes) 000946 1763340384.685181 mapwrite 0x01252f thru 0x01490d (0x23df bytes) 000951 1763340384.728183 mapread 0x013a68 thru 0x013fcb (0x564 bytes) 000952 1763340384.729549 write 0x00ba69 thru 0x017824 (0xbdbc bytes) 000953 1763340384.730907 trunc from 0x03e880 to 0x00bf1f (0x32960 bytes) 000954 1763340384.755830 mapwrite 0x0019bd thru 0x00c4e7 (0xab2b bytes) 000955 1763340384.869107 mapread 0x000a60 thru 0x0062e8 (0x5889 bytes) 000956 1763340384.882762 MIRROR_RESYNC 6 mirrors 000957 1763340385.838153 mapwrite 0x009811 thru 0x012547 (0x8d37 bytes) 000959 1763340386.022397 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x12548): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12548): Mirror 14: 0xb0cf8bf9 (stale) Mirror 15: 0xb0cf8bf9 (stale) Mirror 20: 0xb0cf8bf9 (stale) Mirror 21: 0xb0cf8bf9 (stale) Mirror 22: 0xb0cf8bf9 (stale) CRC-32 checksum value for chunk [0, 0x12548): Mirror 1: 0x94eb5cf0 Mirror 23: 0x94eb5cf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12548: skipped 000963 1763340387.048787 write 0x006aa6 thru 0x00e14a (0x76a5 bytes) 000964 1763340387.149530 read 0x00f519 thru 0x012547 (0x302f bytes) 000965 1763340387.158929 write 0x00e62f thru 0x017891 (0x9263 bytes) 000966 1763340387.167829 read 0x000871 thru 0x00ba95 (0xb225 bytes) 000967 1763340387.180730 read 0x00664f thru 0x00abac (0x455e bytes) 000968 1763340387.181984 write 0x014b13 thru 0x017cf6 (0x31e4 bytes) 000970 1763340387.189621 read 0x00e529 thru 0x014560 (0x6038 bytes) 000971 1763340387.190274 mapread 0x00e303 thru 0x0113fd (0x30fb bytes) 000972 1763340387.191157 mapwrite 0x0345cc thru 0x034fdf (0xa14 bytes) 000973 1763340387.334387 mapread 0x023d52 thru 0x02f069 (0xb318 bytes) 000974 1763340387.335479 write 0x016cbe thru 0x018383 (0x16c6 bytes) 000977 1763340387.337249 trunc from 0x034fe0 to 0x016eaa (0x1e135 bytes) 000979 1763340387.361706 read 0x0010c5 thru 0x00d736 (0xc672 bytes) 000980 1763340387.363332 mapwrite 0x03f220 thru 0x03ffff (0xde0 bytes) 000981 1763340387.460058 write 0x03449f thru 0x036052 (0x1bb4 bytes) 000982 1763340387.461053 write 0x0110be thru 0x01e950 (0xd893 bytes) 000983 1763340387.462191 trunc from 0x040000 to 0x02c80b (0x137f4 bytes) 000984 1763340387.486963 read 0x006a95 thru 0x008058 (0x15c4 bytes) 000987 1763340387.488765 trunc from 0x02c80c to 0x022e5e (0x99ad bytes) 000988 1763340387.508299 mapwrite 0x01077a thru 0x01f5b7 (0xee3e bytes) 000990 1763340387.566236 read 0x00d3ea thru 0x01795c (0xa573 bytes) 000991 1763340387.566971 trunc from 0x022e5f to 0x02f889 (0xca2b bytes) 000993 1763340387.598711 mapread 0x021331 thru 0x02b5f2 (0xa2c2 bytes) 000994 1763340387.631876 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f88a): Mirror 14: 0x5b5a56c (stale) Mirror 20: 0x5b5a56c (stale) Mirror 21: 0x5b5a56c (stale) Mirror 22: 0x5b5a56c (stale) Mirror 23: 0x98875cb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f88a): Mirror 14: 0x5b5a56c (stale) Mirror 20: 0x5b5a56c (stale) Mirror 21: 0x5b5a56c (stale) Mirror 22: 0x5b5a56c (stale) Mirror 23: 0x98875cb3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f88a: skipped 000996 1763340388.107223 mapwrite 0x030f1e thru 0x03b0be (0xa1a1 bytes) 000997 1763340388.200787 mapwrite 0x01c9a7 thru 0x021da2 (0x53fc bytes) 000998 1763340388.251769 read 0x02e49b thru 0x039227 (0xad8d bytes) 001000 1763340388.252845 write 0x029ad3 thru 0x03446f (0xa99d bytes) All operations completed A-OK! PASS 70a (93s) == sanity-flr test 71: check mirror extend parent fid ==== 19:46:33 (1763340393) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 200a: stress test ===================== 19:46:36 (1763340396) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7429312 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3074016 .. failed Extending file size to 7135936 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8319456 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8302208 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8399296 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8575104 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8445984 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8879392 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8972736 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 54908 54909 54910 54911 54912 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) 192.168.204.101@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (76s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:47:52 (1763340472) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3891712 .. Extending file size to 7730496 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5120768 .. failed Extending file size to 7397440 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7882528 .. Extending file size to 8589984 .. Extending file size to 8427520 .. Extending file size to 8752480 .. Extending file size to 8980608 .. Extending file size to 8765792 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8892416 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56990 56991 56992 56994 56996 done 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) 192.168.204.101@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:49:08 (1763340548) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4096 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4096 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11868 s, 17.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.29731 s, 7.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x50615bae (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:49:26 (1763340566) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0790131 s, 26.5 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 19:49:33 (1763340573) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x213:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2a3:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:49:40 (1763340580) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.251604 s, 8.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:49:48 (1763340588) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158342 s, 13.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:49:55 (1763340595) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28305 s, 7.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:50:27 (1763340627) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg401-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105236 s, 10.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:51:23 (1763340683) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg401-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203428 s, 10.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:52:18 (1763340738) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg401-server mds-ost sync done. Creating new pool test_204f oleg401-server: Pool lustre.test_204f created Adding targets to pool oleg401-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg401-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.389451 s, 5.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg401-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg401-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg401-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:53:32 (1763340812) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:53:39 (1763340819) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:53:47 (1763340827) oleg401-server: Pool lustre.test_206 created Adding targets to pool oleg401-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg401-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg401-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg401-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg401-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:54:12 (1763340852) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815587 s, 12.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336088 s, 31.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xbe769a90 Mirror 3: 0xbe769a90 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:54:23 (1763340863) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:54:27 (1763340867) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:54:31 (1763340871) Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.109749 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1b40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2110000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2110000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2110000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xea0000 Read this (4 bytes): BBBB 192.168.204.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg401-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:55:05 (1763340905) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0202535 s, 404 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:55:13 (1763340913) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493257 s, 21.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x238 lmm_fid: [0x200000404:0x238:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x238 lmm_fid: [0x200000404:0x238:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834488 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-16 19:55:14.000000000 -0500 Modify: 2025-11-16 19:55:15.000000000 -0500 Change: 2025-11-16 19:55:16.000000000 -0500 Birth: 2025-11-16 19:55:14.000000000 -0500 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:55:21 (1763340921) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:55:24 (1763340924) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.584616 s, 70.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0559477 s, 73.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 16 19:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 16 19:55 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:55:31 (1763340931) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:55:38 (1763340938) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1503 sec =========== 19:55:47 (1763340947) === sanity-flr: start cleanup 19:55:49 (1763340949) === === sanity-flr: finish cleanup 19:55:55 (1763340955) ===