-----============= acceptance-small: sanity-flr ============----- Thu Jan 22 04:20:42 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg250-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 04:21:07 (1769073667) === oleg250-client.virtnet: executing check_config_client /mnt/lustre oleg250-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg250-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c8506436000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c8506436000.idle_timeout=debug disable quota as required oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 04:21:52 (1769073712) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7577] == sanity-flr test 0a: lfs mirror create with -N option == 04:21:55 (1769073715) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:22:14 (1769073734) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:22:19 (1769073739) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (6s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:22:25 (1769073745) PASS 0d (36s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:23:01 (1769073781) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:23:06 (1769073786) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 04:23:11 (1769073791) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c8506436000.stats=clear osc.lustre-OST0001-osc-ffff9c8506436000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:23:21 (1769073801) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (15s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:23:36 (1769073816) PASS 0j (12s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:23:48 (1769073828) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:23:52 (1769073832) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 04:24:03 (1769073843) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:24:12 (1769073852) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:24:17 (1769073857) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:24:26 (1769073866) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:24:38 (1769073878) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:24:46 (1769073886) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 2.68201 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212107 s, 4.9 MB/s PASS 21 (16s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:25:02 (1769073902) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.48876 s, 27.9 MB/s PASS 22 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:25:14 (1769073914) Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c8506436000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c8506436000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c8506436000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c8506436000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (87s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:26:41 (1769074001) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.44725 s, 11.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c8506436000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c8506436000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:27:30 (1769074050) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c8506436000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c8506436000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c8506436000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c8506436000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (90s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:29:00 (1769074140) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.08737 s, 8.6 MB/s -rw-r--r-- 1 root root 52428800 Jan 22 04:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.613s user 0m0.000s sys 0m0.189s Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c8506436000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c8506436000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.397s user 0m0.000s sys 0m0.168s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c8506436000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c8506436000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.624s user 0m0.000s sys 0m0.200s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg250-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c8506436000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c8506436000.ost_server_uuid in FULL state after 0 sec PASS 33b (103s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:30:43 (1769074243) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:30:46 (1769074246) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:30:50 (1769074250) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 04:30:55 (1769074255) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 04:31:04 (1769074264) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.693091 s, 18.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:31:19 (1769074279) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344959 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268942 s, 39.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:31:27 (1769074287) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:31:36 (1769074296) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00222845 s, 460 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00117581 s, 871 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103583 s, 989 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00266622 s, 384 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 04:31:48 (1769074308) create test files with size 15751821 .. 7dce26ad8e128233dcb1af662936acd9 - b2fa2e19ec22dac4055a16867058ee1b - 4115a0fc469ade7d2a83e0a6d029e29b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre PASS 37 (35s) == sanity-flr test 38: resync ============================ 04:32:23 (1769074343) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (23s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:32:46 (1769074366) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:32:54 (1769074374) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239222 s, 4.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056152 s, 18.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 04:33:05 (1769074385) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155762 s, 13.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.368579 s, 11.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.363188 s, 5.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 04:33:26 (1769074406) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 04:33:30 (1769074410) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:33:34 (1769074414) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590024 s, 17.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 04:33:43 (1769074423) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:33:47 (1769074427) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:33:57 (1769074437) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.296347 s, 35.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:34:11 (1769074451) llite.lustre-ffff9c85053e6000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.368532 s, 28.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9c8506538000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:34:29 (1769074469) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:35:13 (1769074513) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:35:54 (1769074554) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 04:35:58 (1769074558) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.301134 s, 10.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.14726 s, 21.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa5c9160a (stale) Mirror 2: 0xa5c9160a (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 576995146499de81e4ea423eeb931d41 - mirror 1 checksum: 0f7e9d589b1e90a152413246023119d6 - mirror 2 checksum: 576995146499de81e4ea423eeb931d41 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 22 04:36:02 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 22 04:36:02 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 576995146499de81e4ea423eeb931d41 - mirror 1 checksum: 0f7e9d589b1e90a152413246023119d6 - mirror 2 checksum: 0f7e9d589b1e90a152413246023119d6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 22 04:36:02 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 22 04:36:02 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (14s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:36:14 (1769074574) Failing mds1 on oleg250-server Stopping /mnt/lustre-mds1 (opts:) on oleg250-server 04:36:23 (1769074583) shut down facet: mds1 facet_host: oleg250-server facet_failover_host: oleg250-server Failover mds1 to oleg250-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 04:36:50 (1769074610) targets are mounted 04:36:50 (1769074610) facet_failover done oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364013 s, 28.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (52s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:37:07 (1769074627) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:37:11 (1769074631) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:37:16 (1769074636) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00942247 s, 1.7 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:37:25 (1769074645) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:37:30 (1769074650) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:37:35 (1769074655) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:38:04 (1769074684) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (35s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:38:39 (1769074719) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:39:06 (1769074746) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:39:32 (1769074772) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 04:39:42 (1769074782) mirror create pid 52351 mirror split pid 52352 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52351 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4922: 52352 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 04:40:52 (1769074852) Seed set to 4855 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1769074855.675013 mapwrite 0x03023f thru 0x03e623 (0xe3e5 bytes) 000006 1769074856.017112 read 0x02ed48 thru 0x03c9fc (0xdcb5 bytes) 000008 1769074856.021297 mapread 0x0031bd thru 0x0075db (0x441f bytes) 000012 1769074856.061942 read 0x0087f7 thru 0x00cc55 (0x445f bytes) truncating to largest ever: 0x1e307 000013 1769074856.064297 trunc from 0x03e624 to 0x01e306 (0x2031d bytes) 000016 1769074856.138018 mapread 0x00807a thru 0x00e290 (0x6217 bytes) 000017 1769074856.139894 mapwrite 0x02b02b thru 0x0332cf (0x82a5 bytes) 000018 1769074856.351782 mapread 0x023c66 thru 0x02d74c (0x9ae7 bytes) 000019 1769074856.358644 mapread 0x0190a9 thru 0x01b7e2 (0x273a bytes) 000020 1769074856.360108 write 0x0279e8 thru 0x02f05d (0x7676 bytes) 000021 1769074856.366116 mapwrite 0x03bdb3 thru 0x03ffff (0x424d bytes) 000023 1769074856.503464 write 0x03b619 thru 0x03ffff (0x49e7 bytes) 000024 1769074856.507274 mapwrite 0x01b63f thru 0x0235b8 (0x7f7a bytes) 000025 1769074856.596704 mapwrite 0x00fcf8 thru 0x012bf5 (0x2efe bytes) 000026 1769074856.716197 mapread 0x01387a thru 0x021538 (0xdcbf bytes) truncating to largest ever: 0x3591e 000027 1769074856.730753 trunc from 0x040000 to 0x03591d (0xa6e2 bytes) 000029 1769074856.790045 mapwrite 0x01cfb6 thru 0x02ca9d (0xfae8 bytes) 000030 1769074856.881763 trunc from 0x03591e to 0x02cd3a (0x8be3 bytes) 000031 1769074856.899182 mapread 0x0143f5 thru 0x01dcb2 (0x98be bytes) 000034 1769074856.902198 trunc from 0x02cd3b to 0x031a50 (0x4d16 bytes) 000035 1769074856.924207 mapwrite 0x02440e thru 0x033491 (0xf084 bytes) 000036 1769074857.106885 mapwrite 0x02dc5a thru 0x037808 (0x9baf bytes) 000039 1769074857.323235 mapwrite 0x007ef1 thru 0x012ec7 (0xafd7 bytes) 000040 1769074857.522855 MIRROR_EXTEND to 2 mirrors 000042 1769074858.990557 mapwrite 0x024f82 thru 0x02f4df (0xa55e bytes) 000044 1769074859.522070 write 0x006630 thru 0x015393 (0xed64 bytes) 000045 1769074859.559725 read 0x015d6a thru 0x022c8c (0xcf23 bytes) 000046 1769074859.573541 trunc from 0x037809 to 0x02b98a (0xbe7e bytes) 000047 1769074859.637849 read 0x01247c thru 0x01ea91 (0xc616 bytes) 000048 1769074859.644630 MIRROR_RESYNC 2 mirrors 000050 1769074861.385629 write 0x00ed9e thru 0x01a727 (0xb98a bytes) 000051 1769074861.597854 trunc from 0x02b98b to 0x00a332 (0x21658 bytes) 000053 1769074861.643980 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xa333): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa333): Mirror 2: 0x2b77f1a8 (stale) CRC-32 checksum value for chunk [0, 0xa333): Mirror 1: 0x2b77f1a8 Mirror 3: 0x2b77f1a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa333: skipped 000054 1769074862.934327 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000055 1769074863.459418 mapwrite 0x007f53 thru 0x015408 (0xd4b6 bytes) 000057 1769074863.743836 trunc from 0x015409 to 0x02e320 (0x18f18 bytes) 000059 1769074863.785357 read 0x026a94 thru 0x02a228 (0x3795 bytes) 000060 1769074863.797379 mapwrite 0x00a8cb thru 0x011dfe (0x7534 bytes) 000061 1769074863.869804 mapwrite 0x012656 thru 0x01f7be (0xd169 bytes) 000064 1769074864.076378 read 0x01767b thru 0x020ac7 (0x944d bytes) 000065 1769074864.077138 mapwrite 0x005de0 thru 0x0116a0 (0xb8c1 bytes) 000067 1769074864.139302 trunc from 0x02e321 to 0x003b9c (0x2a784 bytes) 000068 1769074864.175213 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000073 1769074864.346797 mapread 0x001101 thru 0x003b9c (0x2a9c bytes) 000075 1769074864.375982 write 0x025986 thru 0x02fe57 (0xa4d2 bytes) 000076 1769074864.379352 read 0x02ee50 thru 0x02fe57 (0x1008 bytes) 000078 1769074864.381930 read 0x007741 thru 0x00aa40 (0x3300 bytes) 000079 1769074864.395646 mapread 0x022de6 thru 0x024391 (0x15ac bytes) 000081 1769074864.397973 mapwrite 0x03ec51 thru 0x03ffff (0x13af bytes) 000083 1769074864.503724 write 0x020a86 thru 0x0264d6 (0x5a51 bytes) 000089 1769074864.519949 mapread 0x017064 thru 0x02507a (0xe017 bytes) truncating to largest ever: 0x35e39 000092 1769074864.527935 trunc from 0x040000 to 0x035e38 (0xa1c7 bytes) 000094 1769074864.580630 read 0x020f2b thru 0x024681 (0x3757 bytes) 000096 1769074864.584362 write 0x037d97 thru 0x03c3a1 (0x460b bytes) 000098 1769074864.587613 mapwrite 0x033ca1 thru 0x03a5d3 (0x6933 bytes) 000101 1769074864.679164 read 0x015b97 thru 0x01fbc2 (0xa02c bytes) 000103 1769074864.682097 trunc from 0x03c3a2 to 0x01a63f (0x21d62 bytes) 000104 1769074864.699247 mapwrite 0x014b34 thru 0x01b468 (0x6935 bytes) 000106 1769074864.798210 write 0x022ff0 thru 0x02b88a (0x889b bytes) 000107 1769074864.799552 read 0x01f5df thru 0x028115 (0x8b37 bytes) 000108 1769074864.812716 write 0x012b5c thru 0x021bee (0xf093 bytes) 000109 1769074864.814072 trunc from 0x02b88b to 0x01b25d (0x1062d bytes) truncating to largest ever: 0x3d8ae 000110 1769074864.839782 trunc from 0x01b25e to 0x03d8ad (0x22650 bytes) 000112 1769074864.864329 mapread 0x008ab7 thru 0x016cc7 (0xe211 bytes) 000115 1769074864.869545 write 0x03900d thru 0x03a2a4 (0x1298 bytes) 000116 1769074864.887820 write 0x0167fc thru 0x024db6 (0xe5bb bytes) 000117 1769074864.896593 read 0x00b0e9 thru 0x01745e (0xc376 bytes) 000118 1769074864.897246 mapwrite 0x008816 thru 0x01756a (0xed55 bytes) 000119 1769074864.976320 mapread 0x028fda thru 0x031ae4 (0x8b0b bytes) 000124 1769074864.994907 mapwrite 0x02b142 thru 0x033318 (0x81d7 bytes) 000126 1769074865.053176 read 0x013afe thru 0x020890 (0xcd93 bytes) 000127 1769074865.053989 mapwrite 0x01e95a thru 0x021601 (0x2ca8 bytes) 000128 1769074865.091896 read 0x00055c thru 0x0061a6 (0x5c4b bytes) 000130 1769074865.092950 mapwrite 0x01daad thru 0x01fe90 (0x23e4 bytes) 000132 1769074865.134203 write 0x005c67 thru 0x013aec (0xde86 bytes) 000133 1769074865.135507 read 0x01ebc7 thru 0x025e88 (0x72c2 bytes) 000134 1769074865.136044 write 0x00bc30 thru 0x012f54 (0x7325 bytes) 000135 1769074865.138235 mapread 0x01c311 thru 0x01e213 (0x1f03 bytes) 000141 1769074865.142694 write 0x03d1b6 thru 0x03ffff (0x2e4a bytes) 000142 1769074865.143758 write 0x011e20 thru 0x01ef06 (0xd0e7 bytes) 000144 1769074865.146261 mapwrite 0x039d5a thru 0x03a79c (0xa43 bytes) 000145 1769074865.302648 write 0x01f80a thru 0x0202a9 (0xaa0 bytes) 000149 1769074865.305400 mapread 0x001f54 thru 0x008798 (0x6845 bytes) 000150 1769074865.306767 write 0x024c20 thru 0x02c3de (0x77bf bytes) 000151 1769074865.308630 read 0x017d6d thru 0x019f55 (0x21e9 bytes) 000155 1769074865.310794 mapwrite 0x023d53 thru 0x032b99 (0xee47 bytes) 000156 1769074865.446265 write 0x035cc8 thru 0x03ffff (0xa338 bytes) 000160 1769074865.462899 read 0x03a48a thru 0x03ffff (0x5b76 bytes) 000162 1769074865.468577 mapread 0x026055 thru 0x031553 (0xb4ff bytes) 000164 1769074865.470095 mapwrite 0x02e0bc thru 0x02f381 (0x12c6 bytes) 000165 1769074865.596816 mapread 0x0278d1 thru 0x028b5e (0x128e bytes) 000167 1769074865.598028 write 0x00c90f thru 0x00ca3d (0x12f bytes) 000168 1769074865.604004 MIRROR_EXTEND to 2 mirrors 000169 1769074866.631941 mapwrite 0x0291f1 thru 0x0387f5 (0xf605 bytes) 000171 1769074866.895835 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1e9264e8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9427f7a5 Mirror 3: 0x9427f7a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000173 1769074868.410025 read 0x025c4b thru 0x02ce85 (0x723b bytes) 000174 1769074868.435305 mapread 0x00f58d thru 0x013b69 (0x45dd bytes) 000175 1769074868.468451 mapread 0x00f575 thru 0x016e46 (0x78d2 bytes) 000177 1769074868.469394 trunc from 0x040000 to 0x00a0f5 (0x35f0a bytes) 000179 1769074868.695312 mapwrite 0x028637 thru 0x037cc0 (0xf68a bytes) 000181 1769074868.815754 trunc from 0x037cc1 to 0x024567 (0x13759 bytes) 000182 1769074868.851847 read 0x01e1ee thru 0x024567 (0x637a bytes) 000184 1769074868.890234 mapread 0x021b8a thru 0x024567 (0x29de bytes) 000186 1769074868.891592 mapwrite 0x008d2a thru 0x013592 (0xa869 bytes) 000187 1769074869.035723 mapwrite 0x00d1b3 thru 0x01aa4f (0xd89d bytes) 000190 1769074869.128153 read 0x015ba4 thru 0x01ffaa (0xa407 bytes) 000191 1769074869.128789 trunc from 0x024568 to 0x000940 (0x23c27 bytes) 000192 1769074869.160927 mapwrite 0x03581f thru 0x039a9e (0x4280 bytes) 000194 1769074869.340741 read 0x0092f3 thru 0x011e30 (0x8b3e bytes) 000195 1769074869.361003 write 0x025e12 thru 0x034aeb (0xecda bytes) 000196 1769074869.362280 trunc from 0x039a9f to 0x00be6c (0x2dc32 bytes) 000197 1769074869.392250 mapwrite 0x001c7c thru 0x0086e1 (0x6a66 bytes) 000198 1769074869.471335 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xbe6d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbe6d): Mirror 2: 0xb0b2bec7 (stale) Mirror 3: 0xb0b2bec7 (stale) CRC-32 checksum value for chunk [0, 0xbe6d): Mirror 1: 0xf2d54f87 Mirror 4: 0xf2d54f87 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbe6d: skipped 000200 1769074870.855432 mapread 0x001073 thru 0x00aae1 (0x9a6f bytes) 000201 1769074870.903564 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbe6d): Mirror 2: 0xb0b2bec7 (stale) Mirror 3: 0xb0b2bec7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbe6d): Mirror 2: 0xb0b2bec7 (stale) Mirror 3: 0xb0b2bec7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbe6d: skipped 000203 1769074871.468814 mapread 0x0081dc thru 0x008e24 (0xc49 bytes) 000204 1769074871.488295 mapwrite 0x022cb8 thru 0x02a1ce (0x7517 bytes) 000205 1769074871.675814 mapwrite 0x01033e thru 0x017850 (0x7513 bytes) 000206 1769074871.766369 mapwrite 0x002362 thru 0x00b642 (0x92e1 bytes) 000209 1769074871.971875 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2a1cf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a1cf): Mirror 2: 0x762f4feb (stale) Mirror 3: 0x8cd2c7af (stale) CRC-32 checksum value for chunk [0, 0x2a1cf): Mirror 1: 0xfa8c0b63 Mirror 4: 0xfa8c0b63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a1cf: skipped 000212 1769074873.019436 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a1cf): Mirror 3: 0x8cd2c7af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a1cf): Mirror 3: 0x8cd2c7af (stale) CRC-32 checksum value for chunk [0, 0x2a1cf): Mirror 1: 0xfa8c0b63 Mirror 4: 0xfa8c0b63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a1cf: skipped 000213 1769074873.802065 mapread 0x027687 thru 0x029e32 (0x27ac bytes) 000214 1769074873.840767 mapwrite 0x02c8c7 thru 0x0375e2 (0xad1c bytes) 000219 1769074874.153295 trunc from 0x0375e3 to 0x02bf24 (0xb6be bytes) 000221 1769074874.211074 read 0x01d632 thru 0x028cd7 (0xb6a6 bytes) 000223 1769074874.248668 mapwrite 0x007382 thru 0x0150a4 (0xdd23 bytes) 000224 1769074874.436065 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2bf25): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bf25): Mirror 3: 0x70db0327 (stale) Mirror 4: 0xebb2af19 (stale) CRC-32 checksum value for chunk [0, 0x2bf25): Mirror 1: 0x1e038f2f Mirror 5: 0x1e038f2f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bf25: skipped 000225 1769074875.931337 read 0x013ec0 thru 0x0189ae (0x4aef bytes) 000228 1769074875.994719 trunc from 0x02bf25 to 0x03abcf (0xecab bytes) 000230 1769074876.155238 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3abd0): Mirror 3: 0x83533059 (stale) Mirror 5: 0x3dbbd53a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3abd0): Mirror 3: 0x83533059 (stale) Mirror 5: 0x3dbbd53a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3abd0: skipped 000232 1769074876.489849 read 0x0059e6 thru 0x0120ff (0xc71a bytes) 000233 1769074876.500160 read 0x015d29 thru 0x01702e (0x1306 bytes) 000234 1769074876.514688 trunc from 0x03abd0 to 0x00bbd1 (0x2effe bytes) 000237 1769074876.535197 write 0x0022eb thru 0x004f2c (0x2c42 bytes) 000238 1769074876.536457 mapread 0x002f99 thru 0x00bbd1 (0x8c39 bytes) 000239 1769074876.542897 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbbd2): Mirror 3: 0x27bfef8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbbd2): Mirror 3: 0x27bfef8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbbd2: skipped 000240 1769074876.832130 read 0x00115f thru 0x005089 (0x3f2b bytes) 000241 1769074876.846099 read 0x009d04 thru 0x00aef4 (0x11f1 bytes) 000244 1769074876.861233 write 0x0380f4 thru 0x03ffff (0x7f0c bytes) 000246 1769074876.862981 mapwrite 0x00eea0 thru 0x01310d (0x426e bytes) 000248 1769074876.995267 mapread 0x035331 thru 0x03ffff (0xaccf bytes) 000249 1769074876.996155 mapread 0x006450 thru 0x0151c8 (0xed79 bytes) 000250 1769074876.996991 mapwrite 0x036ae4 thru 0x03cba0 (0x60bd bytes) 000251 1769074877.075293 mapwrite 0x0385ac thru 0x03ffff (0x7a54 bytes) 000255 1769074877.153333 mapwrite 0x034762 thru 0x03b661 (0x6f00 bytes) 000256 1769074877.229226 trunc from 0x040000 to 0x007b6e (0x38491 bytes) 000260 1769074877.262679 trunc from 0x007b6f to 0x029a2a (0x21ebc bytes) 000261 1769074877.293002 mapread 0x015d13 thru 0x016b12 (0xe00 bytes) 000262 1769074877.318057 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000263 1769074877.509852 write 0x004634 thru 0x006394 (0x1d61 bytes) 000264 1769074877.521461 write 0x02ec74 thru 0x030733 (0x1ac0 bytes) 000267 1769074877.523292 mapread 0x0010d1 thru 0x00fbee (0xeb1e bytes) 000271 1769074877.541105 MIRROR_EXTEND to 2 mirrors 000273 1769074878.647441 MIRROR_RESYNC 2 mirrors 000274 1769074879.035141 mapread 0x02f86c thru 0x030051 (0x7e6 bytes) 000277 1769074879.049540 trunc from 0x030734 to 0x03c9fa (0xc2c7 bytes) 000278 1769074879.145274 read 0x019ed0 thru 0x01c8d5 (0x2a06 bytes) 000280 1769074879.152059 read 0x01462a thru 0x014aca (0x4a1 bytes) 000286 1769074879.167664 mapread 0x00c776 thru 0x00cbab (0x436 bytes) 000288 1769074879.169733 read 0x020ea3 thru 0x023347 (0x24a5 bytes) 000289 1769074879.170636 mapwrite 0x03d536 thru 0x03e1f5 (0xcc0 bytes) 000290 1769074879.265797 trunc from 0x03e1f6 to 0x0002d5 (0x3df20 bytes) 000291 1769074879.310300 MIRROR_RESYNC 2 mirrors 000292 1769074880.061378 trunc from 0x0002d6 to 0x03194b (0x31676 bytes) 000295 1769074880.252879 mapwrite 0x030c30 thru 0x03cde4 (0xc1b5 bytes) 000296 1769074880.393750 read 0x00ca9e thru 0x010979 (0x3edc bytes) 000297 1769074880.404636 mapwrite 0x00e8cf thru 0x011ef8 (0x362a bytes) 000302 1769074880.479218 mapwrite 0x010e09 thru 0x0149fb (0x3bf3 bytes) 000308 1769074880.561653 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cde5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cde5): Mirror 2: 0xcb804636 (stale) CRC-32 checksum value for chunk [0, 0x3cde5): Mirror 1: 0x1b5de1e Mirror 3: 0x1b5de1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cde5: skipped 000309 1769074881.843389 read 0x03c78e thru 0x03cde4 (0x657 bytes) 000310 1769074881.848116 mapwrite 0x0233d8 thru 0x030a2d (0xd656 bytes) 000311 1769074882.067925 trunc from 0x03cde5 to 0x013ce2 (0x29102 bytes) 000312 1769074882.099658 MIRROR_RESYNC 3 mirrors 000314 1769074883.204813 mapread 0x0082b5 thru 0x012c79 (0xa9c5 bytes) 000316 1769074883.229173 read 0x00d797 thru 0x012a72 (0x52dc bytes) 000317 1769074883.229829 read 0x002cb1 thru 0x0044ca (0x181a bytes) 000318 1769074883.242714 write 0x0049a0 thru 0x0114bc (0xcb1d bytes) 000319 1769074883.451628 write 0x01e28e thru 0x025074 (0x6de7 bytes) 000320 1769074883.453429 read 0x018041 thru 0x025074 (0xd034 bytes) 000322 1769074883.480590 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x3fd96 000323 1769074884.624278 trunc from 0x025075 to 0x03fd95 (0x1ad21 bytes) 000326 1769074884.775114 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd96): Mirror 3: 0xe4e06c19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd96): Mirror 3: 0xe4e06c19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd96: skipped 000327 1769074885.259646 mapwrite 0x00255d thru 0x0059db (0x347f bytes) 000328 1769074885.414461 trunc from 0x03fd96 to 0x037e71 (0x7f24 bytes) 000330 1769074885.464826 mapwrite 0x0340e7 thru 0x0386e0 (0x45fa bytes) 000331 1769074885.586100 MIRROR_RESYNC 2 mirrors 000332 1769074886.767519 mapwrite 0x02395e thru 0x02c19f (0x8842 bytes) 000333 1769074887.122143 read 0x02d46f thru 0x037448 (0x9fda bytes) 000334 1769074887.137423 read 0x001843 thru 0x00d410 (0xbbce bytes) 000335 1769074887.172698 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000336 1769074887.442168 write 0x01f24e thru 0x020581 (0x1334 bytes) 000337 1769074887.460743 write 0x000d74 thru 0x00bb4d (0xadda bytes) 000338 1769074887.477049 read 0x03120b thru 0x031d40 (0xb36 bytes) 000339 1769074887.489300 mapread 0x02eadc thru 0x032bf9 (0x411e bytes) 000340 1769074887.506110 read 0x0363a2 thru 0x0386e0 (0x233f bytes) 000343 1769074887.537548 trunc from 0x0386e1 to 0x02d6d9 (0xb007 bytes) 000346 1769074887.566038 trunc from 0x02d6da to 0x0300e9 (0x2a10 bytes) 000348 1769074887.612423 read 0x0008f9 thru 0x000ce4 (0x3ec bytes) 000350 1769074887.620385 write 0x0266ba thru 0x036581 (0xfec8 bytes) 000352 1769074887.623519 write 0x003a65 thru 0x0073fd (0x3999 bytes) 000353 1769074887.628712 write 0x02bbc2 thru 0x0301f3 (0x4632 bytes) 000357 1769074887.634270 write 0x025fd9 thru 0x033f41 (0xdf69 bytes) 000360 1769074887.639062 trunc from 0x036582 to 0x02680e (0xfd73 bytes) 000363 1769074887.685373 read 0x006af5 thru 0x0096a9 (0x2bb5 bytes) 000364 1769074887.685979 read 0x006c5f thru 0x014700 (0xdaa2 bytes) 000365 1769074887.692360 read 0x0219b3 thru 0x023fd0 (0x261e bytes) 000370 1769074887.699807 read 0x01b6d4 thru 0x020c19 (0x5546 bytes) 000371 1769074887.700554 read 0x0020a1 thru 0x00ca46 (0xa9a6 bytes) 000372 1769074887.705876 write 0x00738e thru 0x010ed9 (0x9b4c bytes) 000377 1769074887.714215 write 0x034527 thru 0x038096 (0x3b70 bytes) 000378 1769074887.715300 read 0x01d09d thru 0x0284b9 (0xb41d bytes) 000379 1769074887.738276 read 0x020ddf thru 0x029a57 (0x8c79 bytes) 000381 1769074887.748532 read 0x01e583 thru 0x0248cd (0x634b bytes) 000383 1769074887.752334 mapwrite 0x032af1 thru 0x035cf6 (0x3206 bytes) 000385 1769074887.995155 write 0x00a550 thru 0x010aee (0x659f bytes) 000389 1769074888.007854 mapread 0x00026d thru 0x00263b (0x23cf bytes) 000390 1769074888.008692 mapwrite 0x02f8f0 thru 0x0399cb (0xa0dc bytes) 000391 1769074888.304092 mapread 0x00f3d0 thru 0x01ab83 (0xb7b4 bytes) 000392 1769074888.306941 mapwrite 0x03fb50 thru 0x03ffff (0x4b0 bytes) 000393 1769074888.490601 mapread 0x03726a thru 0x03a109 (0x2ea0 bytes) 000394 1769074888.492365 write 0x02df5e thru 0x02e656 (0x6f9 bytes) 000395 1769074888.502294 mapread 0x03e958 thru 0x03ffff (0x16a8 bytes) 000396 1769074888.510619 read 0x03e116 thru 0x03ffff (0x1eea bytes) 000398 1769074888.527660 MIRROR_EXTEND to 2 mirrors 000399 1769074889.979242 mapread 0x025b90 thru 0x02bf63 (0x63d4 bytes) 000400 1769074890.017684 mapwrite 0x01f369 thru 0x020d6d (0x1a05 bytes) 000402 1769074890.252902 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6c133b57 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdd25f70d Mirror 3: 0xdd25f70d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000404 1769074891.422694 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6c133b57 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdd25f70d Mirror 3: 0xdd25f70d Mirror 4: 0xdd25f70d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000405 1769074892.567628 read 0x01105d thru 0x01ef56 (0xdefa bytes) 000407 1769074892.629389 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6c133b57 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdd25f70d Mirror 3: 0xdd25f70d Mirror 4: 0xdd25f70d Mirror 5: 0xdd25f70d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000408 1769074894.034146 write 0x01e169 thru 0x01eadd (0x975 bytes) 000409 1769074894.178769 mapread 0x00fe68 thru 0x0115cf (0x1768 bytes) 000410 1769074894.204422 mapwrite 0x0386bf thru 0x03ffff (0x7941 bytes) 000413 1769074894.262806 mapwrite 0x02d7d3 thru 0x02e327 (0xb55 bytes) 000414 1769074894.323715 mapread 0x02b397 thru 0x02c4b1 (0x111b bytes) 000415 1769074894.326034 read 0x010229 thru 0x0136f4 (0x34cc bytes) 000417 1769074894.331209 mapread 0x0317e0 thru 0x03a50a (0x8d2b bytes) 000418 1769074894.343393 MIRROR_RESYNC 5 mirrors 000419 1769074895.788076 mapread 0x003e9e thru 0x00a0f4 (0x6257 bytes) 000420 1769074895.809462 mapread 0x02d7d1 thru 0x032060 (0x4890 bytes) 000421 1769074895.820749 trunc from 0x040000 to 0x018861 (0x2779e bytes) 000424 1769074895.922684 MIRROR_RESYNC 5 mirrors 000427 1769074897.109632 mapwrite 0x00f49e thru 0x0132fa (0x3e5d bytes) 000429 1769074897.325784 write 0x03f733 thru 0x03ffff (0x8cd bytes) 000430 1769074897.328378 mapwrite 0x019668 thru 0x01bb82 (0x251b bytes) 000431 1769074897.428860 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2328380c (stale) Mirror 4: 0x2328380c (stale) Mirror 5: 0x2328380c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2328380c (stale) Mirror 4: 0x2328380c (stale) Mirror 5: 0x2328380c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000432 1769074897.942269 mapwrite 0x0356be thru 0x03ffff (0xa942 bytes) 000433 1769074898.023468 MIRROR_RESYNC 4 mirrors 000434 1769074899.078639 write 0x0274e8 thru 0x02d4a1 (0x5fba bytes) 000436 1769074899.152793 trunc from 0x040000 to 0x02673c (0x198c3 bytes) 000439 1769074899.185329 mapread 0x01dee4 thru 0x024dad (0x6eca bytes) 000440 1769074899.198359 mapwrite 0x03d68d thru 0x03ffff (0x2973 bytes) 000441 1769074899.270857 trunc from 0x040000 to 0x00e2db (0x31d24 bytes) 000442 1769074899.291467 read 0x0092e3 thru 0x00e2db (0x4ff9 bytes) 000443 1769074899.300909 mapread 0x000442 thru 0x00a8a4 (0xa463 bytes) 000445 1769074899.317683 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe2dc): Mirror 4: 0x7af58d99 (stale) Mirror 5: 0x7af58d99 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe2dc): Mirror 4: 0x7af58d99 (stale) Mirror 5: 0x7af58d99 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe2dc: skipped 000446 1769074899.643092 mapwrite 0x01d94f thru 0x0280d8 (0xa78a bytes) 000447 1769074899.741562 mapwrite 0x00a0a5 thru 0x010d5d (0x6cb9 bytes) 000448 1769074899.800250 write 0x031799 thru 0x03ddca (0xc632 bytes) 000449 1769074899.802059 trunc from 0x03ddcb to 0x005fe4 (0x37de6 bytes) 000452 1769074899.826752 mapwrite 0x01d199 thru 0x01d980 (0x7e8 bytes) 000454 1769074899.901096 read 0x00cfea thru 0x01a7d6 (0xd7ed bytes) 000456 1769074899.911172 mapwrite 0x01eb7e thru 0x024b31 (0x5fb4 bytes) 000457 1769074899.991825 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24b32): Mirror 5: 0x99ee15bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24b32): Mirror 5: 0x99ee15bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24b32: skipped 000461 1769074900.493937 mapread 0x010c94 thru 0x017134 (0x64a1 bytes) 000463 1769074900.579853 read 0x0097ee thru 0x00f4f0 (0x5d03 bytes) 000464 1769074900.604612 read 0x0145aa thru 0x019a4a (0x54a1 bytes) 000465 1769074900.614109 trunc from 0x024b32 to 0x01e5bd (0x6574 bytes) 000466 1769074900.732453 write 0x03d21e thru 0x03ffff (0x2de2 bytes) 000468 1769074900.735745 trunc from 0x040000 to 0x000c4e (0x3f3b1 bytes) 000469 1769074900.804846 write 0x033d08 thru 0x03d2dc (0x95d5 bytes) 000471 1769074900.806743 write 0x01d9a3 thru 0x026dcb (0x9429 bytes) 000472 1769074900.843538 write 0x020205 thru 0x023efe (0x3cfa bytes) 000475 1769074900.861065 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000477 1769074901.198651 write 0x00c69f thru 0x010156 (0x3ab8 bytes) 000478 1769074901.220819 trunc from 0x03d2dd to 0x02cb57 (0x10785 bytes) 000479 1769074901.261687 read 0x00f779 thru 0x0127a5 (0x302d bytes) 000480 1769074901.272001 write 0x03a111 thru 0x03f5b8 (0x54a8 bytes) 000481 1769074901.273147 trunc from 0x03f5b9 to 0x02fd57 (0xf861 bytes) 000482 1769074901.304225 trunc from 0x02fd58 to 0x02ff99 (0x242 bytes) 000484 1769074901.335157 mapwrite 0x010d26 thru 0x018d0b (0x7fe6 bytes) 000488 1769074901.486649 read 0x02cb70 thru 0x02ff99 (0x342a bytes) 000492 1769074901.517745 trunc from 0x02ff9a to 0x02efe4 (0xfb5 bytes) 000493 1769074901.566351 write 0x01f40a thru 0x02c3a1 (0xcf98 bytes) 000494 1769074901.588517 MIRROR_EXTEND to 2 mirrors 000497 1769074902.560096 mapread 0x00c842 thru 0x01613c (0x98fb bytes) 000498 1769074902.578168 mapread 0x00c7bd thru 0x017309 (0xab4d bytes) 000499 1769074902.590469 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000500 1769074902.737083 write 0x00c2dd thru 0x00c59c (0x2c0 bytes) 000501 1769074902.756205 mapread 0x027ccc thru 0x0293fb (0x1730 bytes) 000502 1769074902.795385 write 0x03063f thru 0x03b813 (0xb1d5 bytes) 000507 1769074902.800833 read 0x027014 thru 0x030ffa (0x9fe7 bytes) 000508 1769074902.819330 write 0x013ed7 thru 0x017d1f (0x3e49 bytes) 000511 1769074902.836961 read 0x02bf80 thru 0x0318e4 (0x5965 bytes) 000512 1769074902.838332 trunc from 0x03b814 to 0x0294f6 (0x1231d bytes) 000513 1769074902.868087 read 0x02726e thru 0x0294f6 (0x2289 bytes) 000516 1769074902.870580 mapread 0x00df1b thru 0x0132d3 (0x53b9 bytes) 000517 1769074902.871912 write 0x00e3c1 thru 0x0118a3 (0x34e3 bytes) 000518 1769074902.874568 mapread 0x025406 thru 0x0294f6 (0x40f1 bytes) 000522 1769074902.881361 read 0x000b95 thru 0x001ad2 (0xf3e bytes) 000523 1769074902.908866 mapwrite 0x0205a5 thru 0x02b9a5 (0xb401 bytes) 000525 1769074903.200677 write 0x01a63a thru 0x01fde8 (0x57af bytes) 000528 1769074903.208174 mapwrite 0x02c7e5 thru 0x03694d (0xa169 bytes) 000529 1769074903.382469 mapread 0x0079b1 thru 0x00ddd2 (0x6422 bytes) 000530 1769074903.385669 read 0x0261b0 thru 0x02f521 (0x9372 bytes) 000532 1769074903.390528 write 0x0306ae thru 0x033642 (0x2f95 bytes) 000533 1769074903.393387 trunc from 0x03694e to 0x03772a (0xddd bytes) 000534 1769074903.430065 trunc from 0x03772b to 0x02d3f9 (0xa331 bytes) 000536 1769074903.505956 mapread 0x02b094 thru 0x02d3f9 (0x2366 bytes) 000537 1769074903.509066 mapread 0x01f703 thru 0x025f9e (0x689c bytes) 000539 1769074903.520119 mapread 0x009dde thru 0x0191b9 (0xf3dc bytes) 000540 1769074903.522622 mapread 0x0017c9 thru 0x00c894 (0xb0cc bytes) 000541 1769074903.539068 mapwrite 0x011bbd thru 0x01d9b0 (0xbdf4 bytes) 000542 1769074903.665214 MIRROR_EXTEND to 2 mirrors 000543 1769074904.687466 write 0x02e8ca thru 0x038bc9 (0xa300 bytes) 000544 1769074904.814979 mapread 0x029c5b thru 0x02ccbd (0x3063 bytes) 000548 1769074904.907554 trunc from 0x038bca to 0x02e549 (0xa680 bytes) 000549 1769074904.949815 write 0x0003a6 thru 0x00ada8 (0xaa03 bytes) 000550 1769074904.978202 trunc from 0x02e54a to 0x030461 (0x1f18 bytes) 000552 1769074905.022760 write 0x021fa8 thru 0x030b34 (0xeb8d bytes) 000553 1769074905.072365 mapread 0x00a90d thru 0x011197 (0x688b bytes) 000554 1769074905.081404 write 0x003e7b thru 0x0113d1 (0xd557 bytes) 000557 1769074905.121921 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30b35): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b35): Mirror 2: 0x7236701b (stale) CRC-32 checksum value for chunk [0, 0x30b35): Mirror 1: 0x81c3652 Mirror 3: 0x81c3652 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b35: skipped 000558 1769074906.791420 trunc from 0x030b35 to 0x01f141 (0x119f3 bytes) 000559 1769074906.959704 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f142): Mirror 2: 0x5a143f25 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f142): Mirror 2: 0x5a143f25 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f142: skipped 000561 1769074907.337403 mapread 0x0096d0 thru 0x00c793 (0x30c4 bytes) 000563 1769074907.374381 write 0x01ce04 thru 0x01d175 (0x372 bytes) 000564 1769074907.388423 write 0x016558 thru 0x024fe3 (0xea8c bytes) 000565 1769074907.392018 write 0x004548 thru 0x009f20 (0x59d9 bytes) 000566 1769074907.395960 trunc from 0x024fe4 to 0x01b3d2 (0x9c11 bytes) 000569 1769074907.418516 read 0x018f15 thru 0x01b3d2 (0x24be bytes) 000571 1769074907.426419 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b3d3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b3d3): Mirror 2: 0xd4772cd3 (stale) CRC-32 checksum value for chunk [0, 0x1b3d3): Mirror 1: 0x6f0dd08a Mirror 3: 0x6f0dd08a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b3d3: skipped 000572 1769074909.041008 mapread 0x00620d thru 0x00caaa (0x689e bytes) 000573 1769074909.113762 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1b3d3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b3d3): Mirror 2: 0xd4772cd3 (stale) CRC-32 checksum value for chunk [0, 0x1b3d3): Mirror 1: 0x6f0dd08a Mirror 3: 0x6f0dd08a Mirror 4: 0x6f0dd08a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b3d3: skipped 000574 1769074910.846113 mapread 0x001f09 thru 0x00863b (0x6733 bytes) 000577 1769074910.909798 trunc from 0x01b3d3 to 0x01b2b5 (0x11d bytes) 000578 1769074911.074049 MIRROR_RESYNC 4 mirrors 000580 1769074912.620779 mapwrite 0x01b3c5 thru 0x021d06 (0x6942 bytes) 000583 1769074913.048782 write 0x004df2 thru 0x00cc4e (0x7e5d bytes) 000584 1769074913.110430 MIRROR_RESYNC 4 mirrors 000585 1769074915.025022 trunc from 0x021d07 to 0x01188f (0x10477 bytes) 000589 1769074915.292124 mapwrite 0x038d1a thru 0x03ffff (0x72e6 bytes) 000591 1769074915.542939 read 0x00dfe0 thru 0x0183be (0xa3df bytes) 000592 1769074915.558085 mapread 0x03d791 thru 0x03ffff (0x286f bytes) 000594 1769074915.559225 mapwrite 0x002808 thru 0x0058cd (0x30c6 bytes) 000595 1769074915.628688 write 0x0028e7 thru 0x00fca3 (0xd3bd bytes) 000596 1769074915.629849 mapwrite 0x03cebd thru 0x03cef7 (0x3b bytes) 000597 1769074915.707882 write 0x030795 thru 0x0396ef (0x8f5b bytes) 000598 1769074915.743717 mapwrite 0x029da7 thru 0x035912 (0xbb6c bytes) 000599 1769074915.876667 mapread 0x0204d1 thru 0x0302aa (0xfdda bytes) 000602 1769074915.896232 read 0x032cab thru 0x03fd47 (0xd09d bytes) 000605 1769074915.905192 mapread 0x023625 thru 0x02913a (0x5b16 bytes) 000606 1769074915.906163 mapread 0x0334d7 thru 0x03ffff (0xcb29 bytes) 000608 1769074915.918239 read 0x01c816 thru 0x029734 (0xcf1f bytes) 000609 1769074915.919385 trunc from 0x040000 to 0x030431 (0xfbce bytes) 000610 1769074915.969207 trunc from 0x030432 to 0x02e886 (0x1bab bytes) 000611 1769074916.014682 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e887): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e887): Mirror 2: 0xb085184e (stale) Mirror 3: 0xb085184e (stale) Mirror 4: 0xb085184e (stale) CRC-32 checksum value for chunk [0, 0x2e887): Mirror 1: 0xf684e87a Mirror 5: 0xf684e87a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e887: skipped 000612 1769074917.900838 read 0x0222cf thru 0x02dc5a (0xb98c bytes) 000614 1769074917.950181 write 0x006755 thru 0x006cba (0x566 bytes) 000615 1769074918.226692 read 0x019d15 thru 0x01b85d (0x1b49 bytes) 000620 1769074918.298083 read 0x0095de thru 0x017e61 (0xe884 bytes) 000621 1769074918.298861 trunc from 0x02e887 to 0x011659 (0x1d22d bytes) 000623 1769074918.395191 mapread 0x0062e6 thru 0x00b932 (0x564d bytes) 000627 1769074918.417038 mapwrite 0x02bebb thru 0x02dfce (0x2114 bytes) 000628 1769074918.662113 mapread 0x01f8ae thru 0x02c124 (0xc877 bytes) 000629 1769074918.675007 write 0x00e8a4 thru 0x01ac1b (0xc378 bytes) 000630 1769074918.678067 mapwrite 0x03559c thru 0x03aeeb (0x5950 bytes) 000631 1769074918.920546 trunc from 0x03aeec to 0x03168f (0x985c bytes) 000633 1769074918.955882 mapread 0x01cc12 thru 0x0269fd (0x9dec bytes) 000635 1769074918.957252 write 0x00562a thru 0x006dcb (0x17a2 bytes) 000639 1769074918.962196 mapread 0x020290 thru 0x02c9c1 (0xc732 bytes) 000641 1769074918.964776 mapread 0x01c390 thru 0x02b25e (0xeecf bytes) 000642 1769074918.966100 mapwrite 0x0054bd thru 0x00d686 (0x81ca bytes) 000643 1769074919.053035 mapread 0x00b73e thru 0x012a0c (0x72cf bytes) 000644 1769074919.065096 MIRROR_RESYNC 5 mirrors 000645 1769074920.351102 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000646 1769074921.088120 trunc from 0x031690 to 0x016f97 (0x1a6f8 bytes) 000648 1769074921.293063 trunc from 0x016f98 to 0x02b885 (0x148ee bytes) 000651 1769074921.353076 trunc from 0x02b886 to 0x027ea4 (0x39e1 bytes) 000652 1769074921.392316 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x27ea5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27ea5): Mirror 3: 0x6e39581d (stale) Mirror 4: 0x6e39581d (stale) Mirror 5: 0x6e39581d (stale) CRC-32 checksum value for chunk [0, 0x27ea5): Mirror 1: 0xd64903a9 Mirror 6: 0xd64903a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27ea5: skipped 000656 1769074922.574127 write 0x033c5a thru 0x03ffff (0xc3a6 bytes) 000657 1769074922.780452 trunc from 0x040000 to 0x00a10d (0x35ef2 bytes) 000661 1769074922.863733 trunc from 0x00a10e to 0x016e7b (0xcd6e bytes) 000663 1769074922.927163 write 0x00dc34 thru 0x00eb7a (0xf47 bytes) 000664 1769074922.999099 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x16e7c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16e7c): Mirror 3: 0xa07a1036 (stale) Mirror 4: 0xa07a1036 (stale) Mirror 5: 0xa07a1036 (stale) Mirror 6: 0xa07a1036 (stale) CRC-32 checksum value for chunk [0, 0x16e7c): Mirror 1: 0x825a7f31 Mirror 7: 0x825a7f31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16e7c: skipped 000667 1769074924.584024 read 0x002e97 thru 0x00a9d5 (0x7b3f bytes) 000669 1769074924.600306 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x16e7c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16e7c): Mirror 3: 0xa07a1036 (stale) Mirror 4: 0xa07a1036 (stale) Mirror 5: 0xa07a1036 (stale) Mirror 6: 0xa07a1036 (stale) CRC-32 checksum value for chunk [0, 0x16e7c): Mirror 1: 0x825a7f31 Mirror 7: 0x825a7f31 Mirror 8: 0x825a7f31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16e7c: skipped 000670 1769074925.994784 read 0x00c2ee thru 0x00c4fa (0x20d bytes) 000671 1769074926.011907 trunc from 0x016e7c to 0x014cf9 (0x2182 bytes) 000673 1769074926.176705 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14cfa): Mirror 3: 0x1ce8f267 (stale) Mirror 5: 0x1ce8f267 (stale) Mirror 6: 0x1ce8f267 (stale) Mirror 7: 0xbce23208 (stale) Mirror 8: 0xbce23208 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14cfa): Mirror 3: 0x1ce8f267 (stale) Mirror 5: 0x1ce8f267 (stale) Mirror 6: 0x1ce8f267 (stale) Mirror 7: 0xbce23208 (stale) Mirror 8: 0xbce23208 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14cfa: skipped 000677 1769074926.820007 mapwrite 0x00d9dd thru 0x01609f (0x86c3 bytes) 000679 1769074927.061644 read 0x00e31f thru 0x012b1c (0x47fe bytes) 000680 1769074927.062283 read 0x00af48 thru 0x01609f (0xb158 bytes) 000681 1769074927.062881 write 0x03a03f thru 0x03ffff (0x5fc1 bytes) 000682 1769074927.066314 read 0x01d449 thru 0x021bb0 (0x4768 bytes) 000683 1769074927.084929 mapwrite 0x027b1b thru 0x02d562 (0x5a48 bytes) 000684 1769074927.198065 write 0x01a417 thru 0x025f59 (0xbb43 bytes) 000685 1769074927.201455 mapread 0x0383dd thru 0x03aafe (0x2722 bytes) 000687 1769074927.202875 write 0x00f1a6 thru 0x0138ec (0x4747 bytes) 000689 1769074927.205954 read 0x031ca1 thru 0x0360fc (0x445c bytes) 000691 1769074927.213207 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5d551cd0 (stale) Mirror 5: 0x5d551cd0 (stale) Mirror 6: 0xcff35307 (stale) Mirror 7: 0xf39b4313 (stale) Mirror 8: 0xf39b4313 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa53f1df9 Mirror 9: 0xa53f1df9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000692 1769074928.424268 trunc from 0x040000 to 0x001cd7 (0x3e328 bytes) 000693 1769074928.583355 mapread 0x00156a thru 0x001cd7 (0x76e bytes) 000694 1769074928.591171 trunc from 0x001cd8 to 0x01f1b0 (0x1d4d9 bytes) 000695 1769074928.612071 mapread 0x009f4a thru 0x019e83 (0xff3a bytes) 000697 1769074928.635619 trunc from 0x01f1b1 to 0x03625b (0x170ab bytes) 000701 1769074928.660110 mapwrite 0x021608 thru 0x026973 (0x536c bytes) 000702 1769074928.732368 trunc from 0x03625c to 0x00f6d9 (0x26b82 bytes) 000704 1769074928.756522 MIRROR_RESYNC 7 mirrors 000706 1769074930.673190 trunc from 0x00f6da to 0x000939 (0xeda0 bytes) 000707 1769074930.775060 write 0x00d10c thru 0x011871 (0x4766 bytes) 000708 1769074930.781079 mapread 0x011155 thru 0x011871 (0x71d bytes) 000709 1769074930.781928 trunc from 0x011872 to 0x01911f (0x78ae bytes) 000710 1769074930.807341 read 0x00a043 thru 0x011b67 (0x7b25 bytes) 000713 1769074930.817863 mapwrite 0x02a773 thru 0x037299 (0xcb27 bytes) 000714 1769074931.144677 trunc from 0x03729a to 0x00a97d (0x2c91c bytes) 000715 1769074931.204149 MIRROR_RESYNC 7 mirrors 000717 1769074933.482209 MIRROR_EXTEND to 8 mirrors 000718 1769074934.573610 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000719 1769074935.191508 mapread 0x0066e6 thru 0x00a97d (0x4298 bytes) 000720 1769074935.302573 read 0x004940 thru 0x00a97d (0x603e bytes) 000721 1769074935.326807 read 0x006cbc thru 0x00a97d (0x3cc2 bytes) 000723 1769074935.405590 mapwrite 0x02f8e9 thru 0x031939 (0x2051 bytes) 000724 1769074935.681992 write 0x00908c thru 0x01198c (0x8901 bytes) 000725 1769074935.683213 read 0x01158f thru 0x01badc (0xa54e bytes) 000729 1769074935.687399 mapread 0x01374a thru 0x01daaa (0xa361 bytes) truncating to largest ever: 0x3fedc 000730 1769074935.689443 trunc from 0x03193a to 0x03fedb (0xe5a2 bytes) 000731 1769074935.718465 trunc from 0x03fedc to 0x015b13 (0x2a3c8 bytes) 000732 1769074935.761247 mapwrite 0x0330c5 thru 0x03ffff (0xcf3b bytes) 000734 1769074935.945726 mapread 0x00c058 thru 0x014d7d (0x8d26 bytes) 000735 1769074935.951005 mapwrite 0x03e4cc thru 0x03ffff (0x1b34 bytes) 000736 1769074936.006668 write 0x031d92 thru 0x03569d (0x390c bytes) 000737 1769074936.009030 write 0x00d402 thru 0x016a6f (0x966e bytes) 000738 1769074936.010523 trunc from 0x040000 to 0x033c20 (0xc3df bytes) 000739 1769074936.040109 mapwrite 0x011331 thru 0x01df94 (0xcc64 bytes) 000740 1769074936.110106 trunc from 0x033c21 to 0x0177e0 (0x1c440 bytes) 000742 1769074936.136329 write 0x031f49 thru 0x03ffff (0xe0b7 bytes) 000744 1769074936.140499 mapwrite 0x013d99 thru 0x016dee (0x3056 bytes) 000745 1769074936.203831 write 0x03275d thru 0x03adb0 (0x8654 bytes) 000748 1769074936.206612 trunc from 0x040000 to 0x00b8f8 (0x34707 bytes) 000749 1769074936.226314 read 0x007af2 thru 0x00b8f8 (0x3e07 bytes) 000750 1769074936.227042 mapread 0x00b2ae thru 0x00b8f8 (0x64b bytes) 000752 1769074936.229254 mapwrite 0x031f96 thru 0x035fbf (0x402a bytes) 000753 1769074936.326970 MIRROR_RESYNC 7 mirrors 000754 1769074937.488171 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000755 1769074938.033049 mapread 0x011f3e thru 0x018873 (0x6936 bytes) 000756 1769074938.056743 write 0x017861 thru 0x026282 (0xea22 bytes) 000758 1769074938.218307 mapwrite 0x038d4a thru 0x03e254 (0x550b bytes) 000760 1769074938.349891 write 0x028416 thru 0x032d9b (0xa986 bytes) 000761 1769074938.362909 write 0x0232c2 thru 0x024fc4 (0x1d03 bytes) 000762 1769074938.365387 trunc from 0x03e255 to 0x026f42 (0x17312 bytes) 000763 1769074938.414903 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x26f43): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26f43): Mirror 3: 0xe36879dc (stale) Mirror 5: 0xe36879dc (stale) Mirror 6: 0xe36879dc (stale) Mirror 7: 0xe36879dc (stale) Mirror 9: 0xe36879dc (stale) CRC-32 checksum value for chunk [0, 0x26f43): Mirror 1: 0x90d08e27 Mirror 10: 0x90d08e27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26f43: skipped 000764 1769074939.727899 mapread 0x00d595 thru 0x00dafa (0x566 bytes) 000765 1769074939.771076 mapwrite 0x0143ff thru 0x015b55 (0x1757 bytes) 000766 1769074939.985421 trunc from 0x026f43 to 0x02ae73 (0x3f31 bytes) 000767 1769074940.004291 mapread 0x011ab5 thru 0x01c2a4 (0xa7f0 bytes) 000768 1769074940.006448 read 0x021beb thru 0x022fd5 (0x13eb bytes) 000769 1769074940.016136 mapread 0x00113e thru 0x001d7b (0xc3e bytes) 000771 1769074940.040150 write 0x001523 thru 0x004310 (0x2dee bytes) 000772 1769074940.041123 read 0x00fe8f thru 0x017b54 (0x7cc6 bytes) 000774 1769074940.058089 MIRROR_RESYNC 7 mirrors 000775 1769074942.061199 mapwrite 0x01324b thru 0x01918f (0x5f45 bytes) 000777 1769074942.321438 mapwrite 0x03ec8d thru 0x03ffff (0x1373 bytes) 000778 1769074942.465454 trunc from 0x040000 to 0x02c685 (0x1397a bytes) 000779 1769074942.502885 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c686): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c686): Mirror 3: 0xa1bed9b9 (stale) Mirror 5: 0xa1bed9b9 (stale) Mirror 6: 0xa1bed9b9 (stale) Mirror 7: 0xa1bed9b9 (stale) Mirror 9: 0xa1bed9b9 (stale) Mirror 10: 0xa1bed9b9 (stale) CRC-32 checksum value for chunk [0, 0x2c686): Mirror 1: 0x9729e3e Mirror 11: 0x9729e3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c686: skipped 000784 1769074944.001762 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c686): Mirror 3: 0xa1bed9b9 (stale) Mirror 5: 0xa1bed9b9 (stale) Mirror 6: 0xa1bed9b9 (stale) Mirror 7: 0xa1bed9b9 (stale) Mirror 9: 0xa1bed9b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c686): Mirror 3: 0xa1bed9b9 (stale) Mirror 5: 0xa1bed9b9 (stale) Mirror 6: 0xa1bed9b9 (stale) Mirror 7: 0xa1bed9b9 (stale) Mirror 9: 0xa1bed9b9 (stale) CRC-32 checksum value for chunk [0, 0x2c686): Mirror 1: 0x9729e3e Mirror 11: 0x9729e3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c686: skipped 000785 1769074944.855759 read 0x004834 thru 0x0088a0 (0x406d bytes) 000787 1769074944.923069 mapwrite 0x017e36 thru 0x01a967 (0x2b32 bytes) 000788 1769074945.210228 mapwrite 0x025c5d thru 0x02848a (0x282e bytes) 000791 1769074945.310055 mapwrite 0x02e03c thru 0x036376 (0x833b bytes) 000793 1769074945.448645 mapread 0x011e89 thru 0x016036 (0x41ae bytes) 000794 1769074945.453462 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36377): Mirror 5: 0xbd40e476 (stale) Mirror 6: 0xbd40e476 (stale) Mirror 7: 0xbd40e476 (stale) Mirror 9: 0xbd40e476 (stale) Mirror 11: 0xa25c419d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36377): Mirror 5: 0xbd40e476 (stale) Mirror 6: 0xbd40e476 (stale) Mirror 7: 0xbd40e476 (stale) Mirror 9: 0xbd40e476 (stale) Mirror 11: 0xa25c419d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36377: skipped 000796 1769074945.965458 mapread 0x009491 thru 0x009e75 (0x9e5 bytes) 000797 1769074945.973159 mapread 0x0262ed thru 0x030010 (0x9d24 bytes) 000798 1769074946.000895 trunc from 0x036377 to 0x03af99 (0x4c23 bytes) 000801 1769074946.023079 mapwrite 0x02a417 thru 0x02a9a1 (0x58b bytes) 000802 1769074946.101470 MIRROR_RESYNC 6 mirrors 000805 1769074947.278292 MIRROR_RESYNC 6 mirrors 000806 1769074947.757454 MIRROR_EXTEND to 7 mirrors 000808 1769074948.647366 read 0x025740 thru 0x02ee7b (0x973c bytes) 000809 1769074948.668838 MIRROR_RESYNC 7 mirrors 000810 1769074949.283569 read 0x0233d9 thru 0x02d97f (0xa5a7 bytes) 000811 1769074949.322287 MIRROR_RESYNC 7 mirrors 000812 1769074950.282048 mapwrite 0x037566 thru 0x03f867 (0x8302 bytes) 000813 1769074950.455501 mapwrite 0x0224fb thru 0x0301cd (0xdcd3 bytes) 000814 1769074950.551803 read 0x0200ed thru 0x02773f (0x7653 bytes) 000815 1769074950.559057 MIRROR_RESYNC 7 mirrors 000816 1769074951.971680 mapwrite 0x00d537 thru 0x010de0 (0x38aa bytes) 000817 1769074952.313736 trunc from 0x03f868 to 0x022b3c (0x1cd2b bytes) 000818 1769074952.368156 trunc from 0x022b3d to 0x03d417 (0x1a8db bytes) 000820 1769074952.409372 trunc from 0x03d418 to 0x039a81 (0x3996 bytes) 000822 1769074952.434028 mapwrite 0x014af7 thru 0x01fb06 (0xb010 bytes) 000823 1769074952.594094 trunc from 0x039a82 to 0x029992 (0x100ef bytes) 000824 1769074952.667597 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x29993): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29993): Mirror 5: 0xbd1a477 (stale) Mirror 6: 0xbd1a477 (stale) Mirror 7: 0xbd1a477 (stale) Mirror 9: 0xbd1a477 (stale) Mirror 11: 0xbd1a477 (stale) Mirror 12: 0xbd1a477 (stale) CRC-32 checksum value for chunk [0, 0x29993): Mirror 1: 0xa569d54d Mirror 13: 0xa569d54d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29993: skipped 000825 1769074954.198873 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29993): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29993): Mirror 5: 0xbd1a477 (stale) Mirror 6: 0xbd1a477 (stale) Mirror 7: 0xbd1a477 (stale) Mirror 9: 0xbd1a477 (stale) Mirror 11: 0xbd1a477 (stale) Mirror 12: 0xbd1a477 (stale) CRC-32 checksum value for chunk [0, 0x29993): Mirror 1: 0xa569d54d Mirror 13: 0xa569d54d Mirror 14: 0xa569d54d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29993: skipped 000829 1769074955.412215 write 0x03722f thru 0x03bfaf (0x4d81 bytes) 000831 1769074955.554915 trunc from 0x03bfb0 to 0x02c44d (0xfb62 bytes) 000832 1769074955.591452 trunc from 0x02c44e to 0x010a89 (0x1b9c4 bytes) 000834 1769074955.624588 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10a8a): Mirror 5: 0x6ae4efb0 (stale) Mirror 6: 0x6ae4efb0 (stale) Mirror 7: 0x6ae4efb0 (stale) Mirror 9: 0x6ae4efb0 (stale) Mirror 11: 0x6ae4efb0 (stale) Mirror 12: 0x6ae4efb0 (stale) Mirror 13: 0x60ac5b45 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10a8a): Mirror 5: 0x6ae4efb0 (stale) Mirror 6: 0x6ae4efb0 (stale) Mirror 7: 0x6ae4efb0 (stale) Mirror 9: 0x6ae4efb0 (stale) Mirror 11: 0x6ae4efb0 (stale) Mirror 12: 0x6ae4efb0 (stale) Mirror 13: 0x60ac5b45 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10a8a: skipped 000836 1769074956.160672 mapwrite 0x03e2b7 thru 0x03ffff (0x1d49 bytes) 000837 1769074956.265443 trunc from 0x040000 to 0x02c904 (0x136fb bytes) 000838 1769074956.308891 MIRROR_RESYNC 8 mirrors 000839 1769074957.956186 trunc from 0x02c905 to 0x016c6f (0x15c95 bytes) 000841 1769074958.066156 mapwrite 0x01a9d8 thru 0x01c45e (0x1a87 bytes) 000843 1769074958.168675 write 0x035e02 thru 0x03ffff (0xa1fe bytes) 000844 1769074958.176078 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 7: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9bd6eb40 Mirror 14: 0x9bd6eb40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000845 1769074959.124433 mapwrite 0x02754e thru 0x02e756 (0x7209 bytes) 000846 1769074959.337851 write 0x029ae6 thru 0x02f071 (0x558c bytes) 000847 1769074959.340111 read 0x02bf61 thru 0x02de66 (0x1f06 bytes) 000849 1769074959.342244 trunc from 0x040000 to 0x027b40 (0x184bf bytes) 000851 1769074959.421762 trunc from 0x027b41 to 0x01f8e9 (0x8257 bytes) 000852 1769074959.444571 mapread 0x01c1c5 thru 0x01f8e9 (0x3725 bytes) 000853 1769074959.458011 mapwrite 0x03bfc7 thru 0x03ffff (0x4039 bytes) 000854 1769074959.604837 mapwrite 0x02a67c thru 0x02e1cc (0x3b51 bytes) 000855 1769074959.696679 mapread 0x02c2db thru 0x033e12 (0x7b38 bytes) 000856 1769074959.702978 read 0x0101f3 thru 0x01ff3b (0xfd49 bytes) 000857 1769074959.714074 write 0x03d268 thru 0x03e457 (0x11f0 bytes) 000859 1769074959.720224 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 7: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) Mirror 14: 0x9bd6eb40 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x245bf45b Mirror 15: 0x245bf45b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000862 1769074960.940863 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 7: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) Mirror 14: 0x9bd6eb40 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x245bf45b Mirror 15: 0x245bf45b Mirror 16: 0x245bf45b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000863 1769074962.263142 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 7: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) Mirror 14: 0x9bd6eb40 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x245bf45b Mirror 15: 0x245bf45b Mirror 16: 0x245bf45b Mirror 17: 0x245bf45b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000864 1769074963.674338 write 0x0213cd thru 0x022523 (0x1157 bytes) 000865 1769074963.856531 write 0x023c11 thru 0x029beb (0x5fdb bytes) 000866 1769074963.876383 write 0x01f3c3 thru 0x025be2 (0x6820 bytes) 000867 1769074963.887176 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 7: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) Mirror 14: 0x9bd6eb40 (stale) Mirror 15: 0x245bf45b (stale) Mirror 17: 0x245bf45b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 7: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) Mirror 14: 0x9bd6eb40 (stale) Mirror 15: 0x245bf45b (stale) Mirror 17: 0x245bf45b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000869 1769074964.627349 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) Mirror 14: 0x9bd6eb40 (stale) Mirror 15: 0x245bf45b (stale) Mirror 17: 0x245bf45b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) Mirror 14: 0x9bd6eb40 (stale) Mirror 15: 0x245bf45b (stale) Mirror 17: 0x245bf45b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000873 1769074965.231855 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) Mirror 14: 0x9bd6eb40 (stale) Mirror 15: 0x245bf45b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) Mirror 14: 0x9bd6eb40 (stale) Mirror 15: 0x245bf45b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000874 1769074965.724011 write 0x017a1a thru 0x01e65e (0x6c45 bytes) 000876 1769074965.736717 mapwrite 0x01a4ac thru 0x021ccf (0x7824 bytes) 000879 1769074965.813833 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdd3f1f53 (stale) Mirror 6: 0xdd3f1f53 (stale) Mirror 9: 0xdd3f1f53 (stale) Mirror 11: 0xdd3f1f53 (stale) Mirror 12: 0xdd3f1f53 (stale) Mirror 13: 0xdd3f1f53 (stale) Mirror 14: 0x9bd6eb40 (stale) Mirror 15: 0x245bf45b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe9eddb90 Mirror 16: 0xe9eddb90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000880 1769074967.270282 trunc from 0x040000 to 0x03f82b (0x7d4 bytes) 000882 1769074967.415465 write 0x03f811 thru 0x03ffff (0x7ef bytes) 000883 1769074967.431830 write 0x0296c8 thru 0x031f3c (0x8875 bytes) 000886 1769074967.452144 mapwrite 0x01a99e thru 0x02787b (0xcede bytes) 000888 1769074967.589535 trunc from 0x040000 to 0x02cd20 (0x132df bytes) 000889 1769074967.623852 MIRROR_RESYNC 10 mirrors 000890 1769074969.171846 mapread 0x002cb2 thru 0x00d60f (0xa95e bytes) 000891 1769074969.194830 MIRROR_EXTEND to 11 mirrors 000892 1769074970.078183 read 0x013fd3 thru 0x015e06 (0x1e34 bytes) 000893 1769074970.084536 mapread 0x0127da thru 0x021c68 (0xf48f bytes) 000895 1769074970.110723 mapread 0x01fc0a thru 0x02cd20 (0xd117 bytes) 000896 1769074970.117684 read 0x025432 thru 0x02cd20 (0x78ef bytes) 000898 1769074970.118331 mapread 0x020465 thru 0x02a2bb (0x9e57 bytes) 000899 1769074970.119923 read 0x024cc9 thru 0x02cd20 (0x8058 bytes) 000900 1769074970.120326 mapread 0x020cae thru 0x025e08 (0x515b bytes) 000901 1769074970.121180 write 0x02a505 thru 0x02cae1 (0x25dd bytes) 000904 1769074970.220238 trunc from 0x02cd21 to 0x01615f (0x16bc1 bytes) 000905 1769074970.247026 write 0x01e220 thru 0x02d6fa (0xf4db bytes) 000911 1769074970.253291 mapread 0x01300c thru 0x01e66d (0xb662 bytes) 000912 1769074970.286789 write 0x007197 thru 0x00f70c (0x8576 bytes) 000913 1769074970.290234 mapwrite 0x01d221 thru 0x02ad6b (0xdb4b bytes) 000915 1769074970.367999 MIRROR_RESYNC 11 mirrors 000916 1769074971.869979 mapread 0x01a247 thru 0x01c3d0 (0x218a bytes) 000917 1769074971.892285 mapread 0x0120a5 thru 0x01aedb (0x8e37 bytes) 000918 1769074971.901776 write 0x00e156 thru 0x012f6a (0x4e15 bytes) 000923 1769074972.026333 mapwrite 0x02aca2 thru 0x036649 (0xb9a8 bytes) 000924 1769074972.278844 write 0x030161 thru 0x03c668 (0xc508 bytes) 000925 1769074972.284856 trunc from 0x03c669 to 0x00bbbd (0x30aab bytes) 000926 1769074972.327912 mapwrite 0x011d83 thru 0x020722 (0xe9a0 bytes) 000934 1769074972.498763 mapwrite 0x023a2a thru 0x02bf51 (0x8528 bytes) 000935 1769074972.616065 mapread 0x0065c3 thru 0x00fee8 (0x9926 bytes) 000939 1769074972.633196 mapread 0x02aad0 thru 0x02bf51 (0x1482 bytes) 000940 1769074972.635792 trunc from 0x02bf52 to 0x01296a (0x195e7 bytes) 000941 1769074972.655250 read 0x00a1a7 thru 0x01296a (0x87c4 bytes) 000942 1769074972.660782 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1296b): Mirror 5: 0x1b48bc14 (stale) Mirror 6: 0x1b48bc14 (stale) Mirror 9: 0x1b48bc14 (stale) Mirror 11: 0x1b48bc14 (stale) Mirror 12: 0x1b48bc14 (stale) Mirror 13: 0x1b48bc14 (stale) Mirror 15: 0x1b48bc14 (stale) Mirror 16: 0x1b48bc14 (stale) Mirror 17: 0x1b48bc14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1296b): Mirror 5: 0x1b48bc14 (stale) Mirror 6: 0x1b48bc14 (stale) Mirror 9: 0x1b48bc14 (stale) Mirror 11: 0x1b48bc14 (stale) Mirror 12: 0x1b48bc14 (stale) Mirror 13: 0x1b48bc14 (stale) Mirror 15: 0x1b48bc14 (stale) Mirror 16: 0x1b48bc14 (stale) Mirror 17: 0x1b48bc14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1296b: skipped 000943 1769074973.182404 write 0x02bf11 thru 0x03ab86 (0xec76 bytes) 000944 1769074973.184992 mapwrite 0x01bedc thru 0x02990e (0xda33 bytes) 000946 1769074973.267146 write 0x030a1d thru 0x03d742 (0xcd26 bytes) 000947 1769074973.278620 MIRROR_RESYNC 10 mirrors 000948 1769074974.749037 read 0x005fa7 thru 0x00eb62 (0x8bbc bytes) 000949 1769074974.757796 mapwrite 0x01e5f6 thru 0x023de6 (0x57f1 bytes) 000950 1769074974.992182 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d743): Mirror 5: 0x531235fd (stale) Mirror 6: 0x531235fd (stale) Mirror 9: 0x531235fd (stale) Mirror 11: 0x531235fd (stale) Mirror 12: 0x531235fd (stale) Mirror 15: 0x531235fd (stale) Mirror 16: 0x531235fd (stale) Mirror 17: 0x531235fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d743): Mirror 5: 0x531235fd (stale) Mirror 6: 0x531235fd (stale) Mirror 9: 0x531235fd (stale) Mirror 11: 0x531235fd (stale) Mirror 12: 0x531235fd (stale) Mirror 15: 0x531235fd (stale) Mirror 16: 0x531235fd (stale) Mirror 17: 0x531235fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d743: skipped 000951 1769074975.488058 mapwrite 0x02ea82 thru 0x032972 (0x3ef1 bytes) 000954 1769074975.557653 read 0x016a10 thru 0x020ea6 (0xa497 bytes) 000959 1769074975.568838 read 0x01488f thru 0x01b5f7 (0x6d69 bytes) 000962 1769074975.580641 mapwrite 0x02dc0c thru 0x038eb5 (0xb2aa bytes) 000963 1769074975.631032 read 0x01f49d thru 0x0201a8 (0xd0c bytes) 000967 1769074975.637665 MIRROR_RESYNC 9 mirrors 000968 1769074977.060359 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) 000969 1769074977.599446 write 0x030816 thru 0x039224 (0x8a0f bytes) 000971 1769074977.690437 write 0x03477f thru 0x036857 (0x20d9 bytes) 000972 1769074977.692953 write 0x02a129 thru 0x032d77 (0x8c4f bytes) 000973 1769074977.704074 trunc from 0x03d743 to 0x02f5d5 (0xe16d bytes) 000976 1769074977.733963 read 0x02330e thru 0x0284df (0x51d2 bytes) 000977 1769074977.734678 mapread 0x010c88 thru 0x01f565 (0xe8de bytes) 000978 1769074977.758356 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2f5d6): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5d6): Mirror 5: 0x7c02b338 (stale) Mirror 9: 0x7c02b338 (stale) Mirror 11: 0x7c02b338 (stale) Mirror 12: 0x7c02b338 (stale) Mirror 15: 0x7c02b338 (stale) Mirror 16: 0x7c02b338 (stale) Mirror 17: 0x7c02b338 (stale) CRC-32 checksum value for chunk [0, 0x2f5d6): Mirror 1: 0x69d2e169 Mirror 18: 0x69d2e169 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5d6: skipped 000979 1769074978.654916 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2f5d6): Mirror 5: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f5d6): Mirror 5: 0x7c02b338 (stale) Mirror 9: 0x7c02b338 (stale) Mirror 11: 0x7c02b338 (stale) Mirror 12: 0x7c02b338 (stale) Mirror 15: 0x7c02b338 (stale) Mirror 16: 0x7c02b338 (stale) Mirror 17: 0x7c02b338 (stale) CRC-32 checksum value for chunk [0, 0x2f5d6): Mirror 1: 0x69d2e169 Mirror 18: 0x69d2e169 Mirror 19: 0x69d2e169 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f5d6: skipped 000980 1769074979.884898 write 0x028b19 thru 0x034e3b (0xc323 bytes) 000981 1769074979.982354 mapread 0x022bd5 thru 0x031c27 (0xf053 bytes) 000982 1769074980.001137 trunc from 0x034e3c to 0x030115 (0x4d26 bytes) 000983 1769074980.027184 read 0x0031c0 thru 0x0098ad (0x66ee bytes) 000984 1769074980.036635 mapwrite 0x03fa35 thru 0x03ffff (0x5cb bytes) 000985 1769074980.127615 mapread 0x0175bc thru 0x020072 (0x8ab7 bytes) 000986 1769074980.139232 write 0x01aa38 thru 0x01d3ce (0x2997 bytes) 000987 1769074980.140852 write 0x015cd9 thru 0x019e9e (0x41c6 bytes) 000988 1769074980.142161 mapwrite 0x00dd6f thru 0x012b72 (0x4e04 bytes) 000990 1769074980.211300 trunc from 0x040000 to 0x005338 (0x3acc7 bytes) 000992 1769074980.239969 MIRROR_RESYNC 10 mirrors 000994 1769074982.127869 mapread 0x003bee thru 0x005338 (0x174b bytes) 000995 1769074982.148402 trunc from 0x005339 to 0x02fb3a (0x2a802 bytes) 000998 1769074982.344854 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x157:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fb3b): Mirror 5: 0x599548a5 (stale) Mirror 9: 0x599548a5 (stale) Mirror 11: 0x599548a5 (stale) Mirror 15: 0x599548a5 (stale) Mirror 16: 0x599548a5 (stale) Mirror 17: 0x599548a5 (stale) Mirror 18: 0x599548a5 (stale) Mirror 19: 0x599548a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb3b): Mirror 5: 0x599548a5 (stale) Mirror 9: 0x599548a5 (stale) Mirror 11: 0x599548a5 (stale) Mirror 15: 0x599548a5 (stale) Mirror 16: 0x599548a5 (stale) Mirror 17: 0x599548a5 (stale) Mirror 18: 0x599548a5 (stale) Mirror 19: 0x599548a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb3b: skipped 001000 1769074983.181090 mapread 0x00399a thru 0x008872 (0x4ed9 bytes) All operations completed A-OK! PASS 70a (137s) == sanity-flr test 71: check mirror extend parent fid ==== 04:43:09 (1769074989) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 04:43:13 (1769074993) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x206:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x202:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.314342 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x206:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x207:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x202:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230715 s, 4.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x206:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x207:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x203:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x202:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x206:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x207:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x203:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x202:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x208:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 04:43:24 (1769075004) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2632672 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6107392 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7274016 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3073760 .. failed Extending file size to 9339392 .. Extending file size to 7185312 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7469088 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7741312 .. Extending file size to 7746848 .. Extending file size to 7969504 .. Extending file size to 8608128 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8619104 .. Extending file size to 9125152 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 55369 55370 55371 55373 55376 done 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) 192.168.202.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:44:50 (1769075090) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3877120 .. Extending file size to 2162496 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6786816 .. failed Extending file size to 6020000 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6899296 .. Extending file size to 7122688 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8600224 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7153152 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7182720 .. Extending file size to 7447040 .. done Extending file size to 8875712 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8743712 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9224704 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57023 57025 57026 57029 57030 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) 192.168.202.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:46:09 (1769075169) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113905 s, 18.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.58446 s, 3.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5ae2ba66 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:46:33 (1769075193) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166782 s, 12.6 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 04:46:42 (1769075202) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1aa:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x219:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x219:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x21d:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:46:51 (1769075211) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.498831 s, 4.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x21e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x21f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (13s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:47:04 (1769075224) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.232088 s, 9.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x220:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x220:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x221:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:47:14 (1769075234) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.29196 s, 7.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x222:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x222:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x223:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 04:47:51 (1769075271) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152863 s, 6.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x225:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x224:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x21f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x225:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x224:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x220:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 04:48:55 (1769075335) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x221:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x222:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.517249 s, 4.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x221:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x222:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x226:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x221:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x222:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x226:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (65s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 04:50:00 (1769075400) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. Creating new pool test_204f oleg250-server: Pool lustre.test_204f created Adding targets to pool oleg250-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg250-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2325 s, 9.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x223:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x227:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x223:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x227:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg250-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg250-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg250-server: Pool lustre.test_204f destroyed PASS 204f (92s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 04:51:32 (1769075492) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 04:51:42 (1769075502) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 04:51:51 (1769075511) oleg250-server: Pool lustre.test_206 created Adding targets to pool oleg250-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg250-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg250-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg250-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg250-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 04:52:22 (1769075542) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118297 s, 8.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685882 s, 15.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2e3d55a1 Mirror 3: 0x2e3d55a1 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 04:52:35 (1769075555) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 04:52:40 (1769075560) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 04:52:44 (1769075564) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.103434 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x14e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb20000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb20000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb20000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1500000 Read this (4 bytes): BBBB 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 04:53:27 (1769075607) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0213544 s, 384 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 04:53:36 (1769075616) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723297 s, 14.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1cf lmm_fid: [0x200000404:0x1cf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1cf lmm_fid: [0x200000404:0x1cf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x231:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834383 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-22 04:53:38.000000000 -0500 Modify: 2026-01-22 04:53:38.000000000 -0500 Change: 2026-01-22 04:53:40.000000000 -0500 Birth: 2026-01-22 04:53:38.000000000 -0500 PASS 210a (13s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 04:53:50 (1769075630) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (5s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 04:53:56 (1769075636) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 1.15563 s, 35.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.069514 s, 58.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 22 04:54 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d1:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 22 04:54 /mnt/lustre/f211.sanity-flr PASS 211 (13s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 04:54:08 (1769075648) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 04:54:17 (1769075657) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 2022 sec =========== 04:54:27 (1769075667) === sanity-flr: start cleanup 04:54:30 (1769075670) === === sanity-flr: finish cleanup 04:54:36 (1769075676) ===