-----============= acceptance-small: sanity-flr ============----- Sun Nov 23 00:36:11 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg413-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 00:36:17 (1763876177) === oleg413-client.virtnet: executing check_config_client /mnt/lustre oleg413-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg413-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f0c86018800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f0c86018800.idle_timeout=debug disable quota as required oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 00:36:27 (1763876187) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8195] == sanity-flr test 0a: lfs mirror create with -N option == 00:36:28 (1763876188) PASS 0a (3s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 00:36:31 (1763876191) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 00:36:32 (1763876192) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 00:36:33 (1763876193) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 00:36:39 (1763876199) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 00:36:40 (1763876200) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 00:36:41 (1763876201) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f0c86018800.stats=clear osc.lustre-OST0001-osc-ffff9f0c86018800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 00:36:43 (1763876203) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 00:36:46 (1763876206) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 00:36:48 (1763876208) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 00:36:49 (1763876209) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 00:36:52 (1763876212) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 00:36:54 (1763876214) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 00:36:56 (1763876216) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 00:36:58 (1763876218) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 00:37:01 (1763876221) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 00:37:03 (1763876223) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.138034 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240292 s, 43.6 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 00:37:05 (1763876225) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.110637 s, 94.8 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 00:37:08 (1763876228) Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0c86018800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0c86018800.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f0c86018800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0c86018800.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (34s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 00:37:42 (1763876262) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0940481 s, 55.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0c86018800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0c86018800.ost_server_uuid in DISCONN state after 3 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 00:37:57 (1763876277) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0c86018800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0c86018800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f0c86018800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0c86018800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:38:23 (1763876303) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.758945 s, 69.1 MB/s -rw-r--r-- 1 root root 52428800 Nov 23 00:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.222s user 0m0.001s sys 0m0.139s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f0c86018800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f0c86018800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.231s user 0m0.000s sys 0m0.147s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f0c86018800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0c86018800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.314s user 0m0.000s sys 0m0.173s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f0c86018800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f0c86018800.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:38:51 (1763876331) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:38:52 (1763876332) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:38:53 (1763876333) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 00:38:54 (1763876334) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 00:38:57 (1763876337) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.154897 s, 81.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:39:00 (1763876340) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125411 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136771 s, 76.7 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:39:03 (1763876343) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:39:06 (1763876346) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00073542 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00066484 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00072314 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000661822 s, 1.5 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 00:39:09 (1763876349) create test files with size 15733809 .. 10f9c20a1efdadc0934976de07b0feb3 - 42ebebf90699a5cc4dba2944d617759a - e1177cbc2f0b18625d5da1727b3000a4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 00:39:15 (1763876355) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:39:20 (1763876360) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:39:22 (1763876362) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415758 s, 25.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x3f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x3f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114818 s, 91.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x40:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0x41:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x40:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 00:39:25 (1763876365) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0332736 s, 63.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.101438 s, 41.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.036328 s, 57.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 00:39:29 (1763876369) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 00:39:30 (1763876370) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:39:31 (1763876371) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147608 s, 71.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 00:39:34 (1763876374) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0921553 s, 34.1 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:39:37 (1763876377) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x49:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x4b:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:39:40 (1763876380) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.108227 s, 96.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:39:59 (1763876399) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9f0ca0409800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.118339 s, 88.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9f0c8533f800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:40:04 (1763876404) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (13s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:40:17 (1763876417) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (12s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:40:29 (1763876429) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 00:40:30 (1763876430) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0822176 s, 38.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0515758 s, 61.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x80947473 (stale) Mirror 2: 0x80947473 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 67643506d5ac9ad0f9e33db0ca81235c - mirror 1 checksum: 697ade1b07d8cfb28eddd14e2413de1c - mirror 2 checksum: 67643506d5ac9ad0f9e33db0ca81235c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 23 00:40:30 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sun Nov 23 00:40:30 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 67643506d5ac9ad0f9e33db0ca81235c - mirror 1 checksum: 697ade1b07d8cfb28eddd14e2413de1c - mirror 2 checksum: 697ade1b07d8cfb28eddd14e2413de1c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 23 00:40:30 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x54:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sun Nov 23 00:40:30 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x55:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 00:40:33 (1763876433) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010648 s, 98.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:40:35 (1763876435) Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 00:40:42 (1763876442) shut down facet: mds1 facet_host: oleg413-server facet_failover_host: oleg413-server Failover mds1 to oleg413-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 00:40:56 (1763876456) targets are mounted 00:40:56 (1763876456) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103079 s, 102 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x57:0x0] } PASS 50A (29s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:41:04 (1763876464) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00922997 s, 2.2 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00520615 s, 3.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:41:07 (1763876467) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00903605 s, 2.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00385931 s, 5.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:41:09 (1763876469) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00253544 s, 6.5 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:41:12 (1763876472) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00894149 s, 2.3 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:41:14 (1763876474) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00225084 s, 1.8 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0100263 s, 409 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:41:16 (1763876476) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:41:22 (1763876482) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:41:41 (1763876501) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x49:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:41:59 (1763876519) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:42:16 (1763876536) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 00:42:18 (1763876538) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 55899 mirror split pid 55900 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 55899 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4919: 55900 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 00:43:20 (1763876600) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6601 000001 1763876601.346061 mapwrite 0x00b05f thru 0x012a55 (0x79f7 bytes) 000002 1763876601.358731 punch from 0x0003fc to 0x006669 (0x626e bytes) fallocating to largest ever: 0x40000 000003 1763876601.359884 fallocate 0x03a21f thru 0x03ffff (0x5de1 bytes) 000005 1763876601.361785 mapwrite 0x02b61b thru 0x033e92 (0x8878 bytes) 000006 1763876601.373061 trunc from 0x033e93 to 0x03f302 (0xb470 bytes) 000007 1763876601.376700 read 0x02df5a thru 0x03ca40 (0xeae7 bytes) 000008 1763876601.379040 punch from 0x033672 to 0x036379 (0x2d08 bytes) 000009 1763876601.380489 punch from 0x00ccaa to 0x01732e (0xa685 bytes) 000010 1763876601.381924 mapwrite 0x0292b0 thru 0x02ea2f (0x5780 bytes) 000011 1763876601.386698 mapread 0x023aca thru 0x02837d (0x48b4 bytes) 000012 1763876601.387188 write 0x001f04 thru 0x00e7ee (0xc8eb bytes) 000013 1763876601.389168 trunc from 0x03f303 to 0x01e59a (0x20d68 bytes) 000014 1763876601.392969 trunc from 0x01e59b to 0x027426 (0x8e8c bytes) 000015 1763876601.396400 read 0x00edaf thru 0x01d53a (0xe78c bytes) 000016 1763876601.398729 write 0x027c28 thru 0x030c3e (0x9017 bytes) 000017 1763876601.399440 punch from 0x00f707 to 0x01399a (0x4294 bytes) 000018 1763876601.400733 mapwrite 0x026dbf thru 0x028be2 (0x1e24 bytes) 000019 1763876601.405294 trunc from 0x030c3f to 0x0023f4 (0x2e84a bytes) 000020 1763876601.408919 mapread 0x00123e thru 0x0023f4 (0x11b7 bytes) 000021 1763876601.409401 write 0x02e162 thru 0x02e91b (0x7ba bytes) 000022 1763876601.409964 fallocate 0x03e9d0 thru 0x03ffff (0x1630 bytes) 000023 1763876601.411446 fallocate 0x00a5f1 thru 0x00aebb (0x8cb bytes) 000024 1763876601.412762 trunc from 0x0181e6 to 0x01900e (0xe29 bytes) 000026 1763876601.414087 trunc from 0x016904 to 0x01ab7c (0x4279 bytes) 000027 1763876601.415307 punch from 0x00670c to 0x008f0d (0x2802 bytes) 000028 1763876601.416423 mapread 0x0274ea thru 0x0322f6 (0xae0d bytes) 000029 1763876601.419729 mapread 0x00e6ff thru 0x017994 (0x9296 bytes) 000030 1763876601.420182 read 0x0308b8 thru 0x038420 (0x7b69 bytes) 000031 1763876601.420497 read 0x000a06 thru 0x007427 (0x6a22 bytes) 000032 1763876601.420797 write 0x01e417 thru 0x026b31 (0x871b bytes) 000033 1763876601.421431 mapread 0x006bdf thru 0x008c3c (0x205e bytes) 000034 1763876601.421842 fallocate 0x0132cf thru 0x014487 (0x11b9 bytes) 000035 1763876601.423124 read 0x03b7dc thru 0x03ffff (0x4824 bytes) 000037 1763876601.423651 punch from 0x015dc7 to 0x01bac8 (0x5d02 bytes) 000038 1763876601.424902 trunc from 0x040000 to 0x00d2d0 (0x32d2f bytes) 000039 1763876601.428815 write 0x023ce8 thru 0x02d3c4 (0x96dd bytes) 000040 1763876601.429641 read 0x004592 thru 0x0061d0 (0x1c3f bytes) 000041 1763876601.431063 MIRROR_EXTEND to 2 mirrors 000042 1763876601.563358 trunc from 0x02d3c5 to 0x006870 (0x26b54 bytes) 000043 1763876601.578553 punch from 0x005b0f to 0x006870 (0xd62 bytes) 000044 1763876601.580027 read 0x0033ef thru 0x006870 (0x3482 bytes) 000045 1763876601.582011 write 0x036e98 thru 0x03c923 (0x5a8c bytes) 000046 1763876601.584489 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c924): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c924): Mirror 2: 0x3b165d91 (stale) CRC-32 checksum value for chunk [0, 0x3c924): Mirror 1: 0x9c8c6697 Mirror 3: 0x9c8c6697 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c924: skipped 000047 1763876601.772213 punch from 0x034f1b to 0x03c923 (0x7a09 bytes) 000048 1763876601.782666 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c924): Mirror 2: 0x3b165d91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c924): Mirror 2: 0x3b165d91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c924: skipped 000049 1763876601.846097 read 0x020bc2 thru 0x0303f6 (0xf835 bytes) 000050 1763876601.848525 fallocate 0x03e408 thru 0x03ffff (0x1bf8 bytes) 000051 1763876601.850012 trunc from 0x02b787 to 0x02cf15 (0x178f bytes) 000052 1763876601.851405 fallocate 0x0352fc thru 0x0386ed (0x33f2 bytes) 000053 1763876601.854350 MIRROR_RESYNC 2 mirrors 000054 1763876602.008718 read 0x0214d5 thru 0x02c3ad (0xaed9 bytes) 000055 1763876602.011227 punch from 0x010ad8 to 0x01e92d (0xde56 bytes) 000056 1763876602.019575 trunc from 0x000e03 to 0x00cb8b (0xbd89 bytes) 000057 1763876602.020858 fallocate 0x0047ed thru 0x01123f (0xca53 bytes) 000058 1763876602.023478 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9b092721 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 3: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000062 1763876602.220717 mapread 0x006c6a thru 0x0073fd (0x794 bytes) 000063 1763876602.228271 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9b092721 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9b092721 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000064 1763876602.293110 read 0x005d67 thru 0x00dca6 (0x7f40 bytes) 000065 1763876602.295741 fallocate 0x015556 thru 0x025243 (0xfcee bytes) 000066 1763876602.299560 punch from 0x02c218 to 0x0374ec (0xb2d5 bytes) 000067 1763876602.300919 fallocate 0x005a44 thru 0x00be9d (0x645a bytes) 000068 1763876602.302330 fallocate 0x018da1 thru 0x01bbc2 (0x2e22 bytes) 000069 1763876602.303735 read 0x01a101 thru 0x0230c7 (0x8fc7 bytes) 000070 1763876602.307871 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9b092721 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 3: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000071 1763876602.509150 punch from 0x00d27b to 0x019899 (0xc61f bytes) 000072 1763876602.518705 read 0x01aee1 thru 0x021739 (0x6859 bytes) 000073 1763876602.521315 fallocate 0x02213c thru 0x032057 (0xff1c bytes) 000074 1763876602.523122 mapwrite 0x036f6e thru 0x039b96 (0x2c29 bytes) 000075 1763876602.531562 MIRROR_RESYNC 3 mirrors 000076 1763876602.716199 fallocate 0x005e72 thru 0x0087ab (0x293a bytes) 000077 1763876602.724181 fallocate 0x03d54b thru 0x03ffff (0x2ab5 bytes) 000078 1763876602.726034 mapread 0x02bf3a thru 0x03a7a6 (0xe86d bytes) 000079 1763876602.729267 trunc from 0x005403 to 0x010bac (0xb7aa bytes) 000080 1763876602.730811 mapread 0x0343cb thru 0x03f56b (0xb1a1 bytes) 000081 1763876602.731477 mapread 0x0060c0 thru 0x00b573 (0x54b4 bytes) 000082 1763876602.734359 trunc from 0x00bb70 to 0x00e51e (0x29af bytes) 000084 1763876602.736060 read 0x02a78c thru 0x0300a1 (0x5916 bytes) 000085 1763876602.736500 trunc from 0x02574b to 0x02da0e (0x82c4 bytes) 000086 1763876602.738006 mapwrite 0x034ecd thru 0x03b308 (0x643c bytes) 000087 1763876602.744087 trunc from 0x02fc58 to 0x03570c (0x5ab5 bytes) 000090 1763876602.746233 trunc from 0x025b03 to 0x0340a7 (0xe5a5 bytes) 000091 1763876602.747542 trunc from 0x01f543 to 0x02b92f (0xc3ed bytes) 000094 1763876602.749270 trunc from 0x03acc3 to 0x03ffff (0x533d bytes) 000095 1763876602.750752 trunc from 0x01fdb6 to 0x0286f5 (0x8940 bytes) 000096 1763876602.751973 mapwrite 0x0024ab thru 0x00b8e5 (0x943b bytes) 000097 1763876602.759713 fallocate 0x03783d thru 0x03ffff (0x87c3 bytes) 000098 1763876602.761221 mapwrite 0x00b69d thru 0x0173cd (0xbd31 bytes) 000099 1763876602.767339 trunc from 0x008455 to 0x00f74a (0x72f6 bytes) 000101 1763876602.768909 mapwrite 0x02d7c2 thru 0x03235e (0x4b9d bytes) 000102 1763876602.773451 read 0x0118de thru 0x01979c (0x7ebf bytes) 000104 1763876602.774265 punch from 0x03e453 to 0x03ffff (0x1bad bytes) 000105 1763876602.775879 write 0x02a175 thru 0x02a8a5 (0x731 bytes) 000106 1763876602.776626 trunc from 0x040000 to 0x00cf6c (0x33093 bytes) 000107 1763876602.787655 MIRROR_RESYNC 3 mirrors 000108 1763876602.942305 punch from 0x007462 to 0x00cf6c (0x5b0b bytes) 000109 1763876602.951227 trunc from 0x00cf6d to 0x0247a5 (0x17839 bytes) 000110 1763876602.959805 write 0x02f59e thru 0x03b1f3 (0xbc56 bytes) 000111 1763876602.961003 read 0x025d77 thru 0x02d629 (0x78b3 bytes) 000112 1763876602.963284 read 0x009f2d thru 0x00fbd2 (0x5ca6 bytes) 000114 1763876602.966468 mapwrite 0x0188a5 thru 0x01db04 (0x5260 bytes) 000115 1763876602.971504 write 0x02d3da thru 0x039afc (0xc723 bytes) 000116 1763876602.972358 write 0x02ddde thru 0x03b897 (0xdaba bytes) 000118 1763876602.973451 mapread 0x0325e7 thru 0x037cd8 (0x56f2 bytes) 000119 1763876602.973944 mapread 0x00f4c0 thru 0x0128dc (0x341d bytes) 000120 1763876602.974414 fallocate 0x01531c thru 0x018cdf (0x39c4 bytes) 000121 1763876602.976012 write 0x0397a9 thru 0x03ffff (0x6857 bytes) 000122 1763876602.976693 write 0x01a45b thru 0x01c5bc (0x2162 bytes) 000123 1763876602.977186 trunc from 0x040000 to 0x01ff50 (0x200af bytes) 000124 1763876602.983337 trunc from 0x009eb1 to 0x017072 (0xd1c2 bytes) 000125 1763876602.984765 write 0x01fe20 thru 0x021ec3 (0x20a4 bytes) 000126 1763876602.985315 fallocate 0x0220d4 thru 0x02a8b4 (0x87e1 bytes) 000127 1763876602.986657 mapwrite 0x0367c9 thru 0x03f30c (0x8b44 bytes) 000129 1763876602.997665 fallocate 0x01538b thru 0x020225 (0xae9b bytes) 000130 1763876602.999071 mapread 0x0206c9 thru 0x02e9e5 (0xe31d bytes) 000131 1763876602.999592 mapread 0x0016cc thru 0x00b6a8 (0x9fdd bytes) 000132 1763876603.000039 write 0x014f2b thru 0x01c05a (0x7130 bytes) 000133 1763876603.000688 fallocate 0x032e7a thru 0x039389 (0x6510 bytes) 000134 1763876603.002049 trunc from 0x03f30d to 0x0281c7 (0x17145 bytes) 000135 1763876603.005553 mapwrite 0x03b5c2 thru 0x03ffff (0x4a3e bytes) 000136 1763876603.016213 write 0x01b63a thru 0x01ece3 (0x36aa bytes) 000137 1763876603.016959 mapread 0x0386fc thru 0x03f637 (0x6f3c bytes) 000138 1763876603.017584 punch from 0x00d4d9 to 0x011b1f (0x4647 bytes) 000139 1763876603.019203 fallocate 0x004d91 thru 0x0125e5 (0xd855 bytes) 000140 1763876603.020708 punch from 0x017f3b to 0x01d440 (0x5506 bytes) 000141 1763876603.023553 mapread 0x012a66 thru 0x01e336 (0xb8d1 bytes) 000143 1763876603.026224 trunc from 0x040000 to 0x016be7 (0x29418 bytes) 000144 1763876603.030557 mapwrite 0x0069f5 thru 0x011dce (0xb3da bytes) 000146 1763876603.038505 write 0x028ce8 thru 0x033c0f (0xaf28 bytes) 000147 1763876603.039859 mapwrite 0x0132b9 thru 0x01840d (0x5155 bytes) 000148 1763876603.048669 mapread 0x025978 thru 0x02cf38 (0x75c1 bytes) 000149 1763876603.049362 trunc from 0x033c10 to 0x031703 (0x250c bytes) 000150 1763876603.053872 mapread 0x00bc58 thru 0x00bcef (0x98 bytes) 000151 1763876603.054397 mapwrite 0x011632 thru 0x017938 (0x6307 bytes) 000152 1763876603.062301 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31704): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31704): Mirror 2: 0x4a2a55b5 (stale) Mirror 3: 0x4a2a55b5 (stale) CRC-32 checksum value for chunk [0, 0x31704): Mirror 1: 0xa70abdaa Mirror 4: 0xa70abdaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31704: skipped 000153 1763876603.278801 read 0x010738 thru 0x01379c (0x3065 bytes) 000154 1763876603.282319 write 0x0278f2 thru 0x031943 (0xa052 bytes) 000155 1763876603.295546 fallocate 0x01667a thru 0x024c8f (0xe616 bytes) 000157 1763876603.299167 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31944): Mirror 3: 0x7c716011 (stale) Mirror 4: 0x98455596 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31944): Mirror 3: 0x7c716011 (stale) Mirror 4: 0x98455596 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31944: skipped 000158 1763876603.390825 write 0x032df8 thru 0x03ffff (0xd208 bytes) 000159 1763876603.393495 fallocate 0x027dda thru 0x034a77 (0xcc9e bytes) 000160 1763876603.395646 read 0x01dbc5 thru 0x01e74a (0xb86 bytes) 000161 1763876603.397766 mapwrite 0x03bf36 thru 0x03ffff (0x40ca bytes) 000162 1763876603.403406 mapread 0x03177b thru 0x03260d (0xe93 bytes) 000163 1763876603.406494 trunc from 0x040000 to 0x030d18 (0xf2e7 bytes) 000164 1763876603.414893 trunc from 0x01da3d to 0x02758e (0x9b52 bytes) 000165 1763876603.416498 mapread 0x023232 thru 0x030658 (0xd427 bytes) 000166 1763876603.419493 mapwrite 0x02be79 thru 0x03a50a (0xe692 bytes) 000167 1763876603.433857 mapwrite 0x01b14b thru 0x022655 (0x750b bytes) 000168 1763876603.439986 mapwrite 0x034bc0 thru 0x03adfe (0x623f bytes) 000169 1763876603.450114 mapread 0x03101c thru 0x03adfe (0x9de3 bytes) 000170 1763876603.450900 punch from 0x011002 to 0x012b19 (0x1b18 bytes) 000171 1763876603.452597 write 0x016e78 thru 0x02185b (0xa9e4 bytes) 000172 1763876603.455301 MIRROR_RESYNC 3 mirrors 000173 1763876603.648148 read 0x01217e thru 0x019315 (0x7198 bytes) 000174 1763876603.651614 MIRROR_RESYNC 3 mirrors 000175 1763876603.753027 read 0x02e114 thru 0x039826 (0xb713 bytes) 000176 1763876603.756388 MIRROR_RESYNC 3 mirrors 000177 1763876603.862753 read 0x025bec thru 0x031bba (0xbfcf bytes) 000178 1763876603.865036 fallocate 0x03698f thru 0x03ffff (0x9671 bytes) 000179 1763876603.873735 trunc from 0x01729b to 0x0269d5 (0xf73b bytes) 000180 1763876603.875146 trunc from 0x03adff to 0x017c45 (0x231b9 bytes) 000181 1763876603.883502 mapread 0x00b5d4 thru 0x017c45 (0xc672 bytes) 000182 1763876603.886794 mapwrite 0x027328 thru 0x02ae14 (0x3aed bytes) 000183 1763876603.896294 read 0x017fa7 thru 0x020b37 (0x8b91 bytes) 000184 1763876603.896701 trunc from 0x02569a to 0x02ae14 (0x577b bytes) 000185 1763876603.898266 trunc from 0x02ae15 to 0x0261ff (0x4c15 bytes) 000186 1763876603.901626 mapwrite 0x03283f thru 0x035d03 (0x34c5 bytes) 000187 1763876603.912160 trunc from 0x017693 to 0x0259e4 (0xe352 bytes) 000189 1763876603.915729 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35d04): Mirror 4: 0x758c7035 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35d04): Mirror 4: 0x758c7035 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35d04: skipped 000190 1763876603.984085 mapwrite 0x03a59d thru 0x03ffff (0x5a63 bytes) 000191 1763876603.995031 mapwrite 0x01f494 thru 0x029cf2 (0xa85f bytes) 000192 1763876604.002386 mapwrite 0x03804d thru 0x03f9b0 (0x7964 bytes) 000193 1763876604.007266 fallocate 0x012627 thru 0x012f4a (0x924 bytes) 000194 1763876604.008843 fallocate 0x01bb08 thru 0x01cad2 (0xfcb bytes) 000195 1763876604.010377 write 0x024f39 thru 0x025d6f (0xe37 bytes) 000196 1763876604.010955 mapwrite 0x03ef14 thru 0x03f00e (0xfb bytes) 000197 1763876604.015166 mapwrite 0x028a3b thru 0x02f5e2 (0x6ba8 bytes) 000198 1763876604.020443 mapwrite 0x02c642 thru 0x039d62 (0xd721 bytes) 000199 1763876604.027670 trunc from 0x040000 to 0x01b004 (0x24ffb bytes) 000200 1763876604.032879 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1b005): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b005): Mirror 4: 0x2593c3e3 (stale) CRC-32 checksum value for chunk [0, 0x1b005): Mirror 1: 0x583cef25 Mirror 5: 0x583cef25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b005: skipped 000201 1763876604.239660 punch from 0x00f35f to 0x01b004 (0xbca6 bytes) 000203 1763876604.249403 trunc from 0x006c90 to 0x0169e4 (0xfd55 bytes) 000204 1763876604.251158 fallocate 0x0260b0 thru 0x027b51 (0x1aa2 bytes) 000205 1763876604.253694 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x27b52): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b52): Mirror 4: 0x9f2bbd6d (stale) Mirror 5: 0xa45afb4f (stale) CRC-32 checksum value for chunk [0, 0x27b52): Mirror 1: 0x1f40c4a9 Mirror 6: 0x1f40c4a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b52: skipped 000206 1763876604.463988 mapwrite 0x0042e3 thru 0x00c6c6 (0x83e4 bytes) 000207 1763876604.500645 trunc from 0x027b52 to 0x01f9ed (0x8164 bytes) 000209 1763876604.506159 read 0x00bf82 thru 0x00c8b2 (0x931 bytes) 000210 1763876604.506555 read 0x00503f thru 0x00894a (0x390c bytes) 000211 1763876604.506852 fallocate 0x00d407 thru 0x019fc3 (0xcbbd bytes) 000212 1763876604.508277 write 0x027bd2 thru 0x028b34 (0xf63 bytes) 000213 1763876604.508790 trunc from 0x028b35 to 0x01f297 (0x989d bytes) 000215 1763876604.512813 read 0x000cfd thru 0x00c90e (0xbc12 bytes) 000216 1763876604.513234 mapwrite 0x01fc55 thru 0x0250bf (0x546b bytes) 000217 1763876604.523763 read 0x017c30 thru 0x01fad0 (0x7ea1 bytes) 000218 1763876604.524372 mapwrite 0x021ad0 thru 0x02af5d (0x948e bytes) 000219 1763876604.535331 punch from 0x00509a to 0x00e406 (0x936d bytes) 000220 1763876604.537112 trunc from 0x010e04 to 0x016b81 (0x5d7e bytes) 000221 1763876604.538698 fallocate 0x00e77f thru 0x018198 (0x9a1a bytes) 000222 1763876604.540399 trunc from 0x02af5e to 0x021955 (0x9608 bytes) 000223 1763876604.544449 mapread 0x019f78 thru 0x01da5a (0x3ae3 bytes) 000224 1763876604.545157 fallocate 0x034f01 thru 0x03e491 (0x9591 bytes) 000225 1763876604.548140 MIRROR_RESYNC 4 mirrors 000226 1763876604.758827 read 0x03834e thru 0x03e491 (0x6144 bytes) 000227 1763876604.761209 fallocate 0x0303d1 thru 0x03f17f (0xedaf bytes) 000229 1763876604.770443 punch from 0x01a27a to 0x025d5b (0xbae2 bytes) 000230 1763876604.772076 fallocate 0x00020d thru 0x007a8b (0x787f bytes) 000231 1763876604.773659 mapwrite 0x02efd2 thru 0x03a8d0 (0xb8ff bytes) 000232 1763876604.782543 trunc from 0x03f180 to 0x02c690 (0x12aef bytes) 000233 1763876604.792806 mapwrite 0x026afd thru 0x02b83c (0x4d40 bytes) 000234 1763876604.798045 fallocate 0x00246a thru 0x00fa87 (0xd61e bytes) 000235 1763876604.800114 mapread 0x00597c thru 0x01028a (0xa90f bytes) 000236 1763876604.800727 punch from 0x023b4a to 0x02c690 (0x8b47 bytes) 000237 1763876604.802338 read 0x000740 thru 0x00bca4 (0xb565 bytes) 000238 1763876604.802695 trunc from 0x024aaf to 0x02c690 (0x7be2 bytes) 000239 1763876604.805320 MIRROR_RESYNC 4 mirrors 000240 1763876605.013229 fallocate 0x00d864 thru 0x014231 (0x69ce bytes) 000242 1763876605.023311 trunc from 0x02c691 to 0x02eadc (0x244c bytes) 000243 1763876605.032581 read 0x01e93b thru 0x0249ca (0x6090 bytes) 000244 1763876605.036905 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2eadd): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eadd): Mirror 4: 0x4edd457b (stale) Mirror 5: 0x4edd457b (stale) Mirror 6: 0x4edd457b (stale) CRC-32 checksum value for chunk [0, 0x2eadd): Mirror 1: 0x4edd457b Mirror 7: 0x4edd457b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eadd: skipped 000245 1763876605.254381 mapread 0x007fa0 thru 0x013352 (0xb3b3 bytes) 000247 1763876605.262403 fallocate 0x03ebc7 thru 0x03ffff (0x1439 bytes) 000248 1763876605.273690 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x32f17272 (stale) Mirror 5: 0x32f17272 (stale) Mirror 6: 0x32f17272 (stale) Mirror 7: 0x32f17272 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x32f17272 Mirror 8: 0x32f17272 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000249 1763876605.476379 punch from 0x025294 to 0x029794 (0x4501 bytes) 000250 1763876605.486643 trunc from 0x001f17 to 0x010c3c (0xed26 bytes) 000251 1763876605.488168 mapwrite 0x02ccdc thru 0x02dc49 (0xf6e bytes) 000252 1763876605.494714 mapread 0x033484 thru 0x03546a (0x1fe7 bytes) 000253 1763876605.495216 trunc from 0x040000 to 0x01459f (0x2ba60 bytes) 000254 1763876605.507044 punch from 0x008e60 to 0x009541 (0x6e2 bytes) 000255 1763876605.508800 mapread 0x005436 thru 0x00b72c (0x62f7 bytes) 000256 1763876605.512434 trunc from 0x0145a0 to 0x02c34d (0x17dae bytes) 000257 1763876605.517013 read 0x018f67 thru 0x01ead7 (0x5b71 bytes) 000258 1763876605.519994 write 0x0162c6 thru 0x025e2e (0xfb69 bytes) 000259 1763876605.520962 mapread 0x0058d5 thru 0x00eb98 (0x92c4 bytes) 000260 1763876605.521683 trunc from 0x02c34e to 0x004c07 (0x27746 bytes) 000261 1763876605.526629 trunc from 0x004c08 to 0x029c15 (0x2500e bytes) 000262 1763876605.532670 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29c16): Mirror 4: 0x25e072ae (stale) Mirror 5: 0x25e072ae (stale) Mirror 6: 0x25e072ae (stale) Mirror 7: 0x25e072ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c16): Mirror 4: 0x25e072ae (stale) Mirror 5: 0x25e072ae (stale) Mirror 6: 0x25e072ae (stale) Mirror 7: 0x25e072ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c16: skipped 000263 1763876605.620415 trunc from 0x017aef to 0x024a3f (0xcf51 bytes) 000264 1763876605.622290 mapread 0x00b007 thru 0x01af8e (0xff88 bytes) 000265 1763876605.625583 write 0x0044ed thru 0x0090d4 (0x4be8 bytes) 000266 1763876605.629164 mapwrite 0x01529f thru 0x01e531 (0x9293 bytes) 000267 1763876605.635190 write 0x026775 thru 0x031486 (0xad12 bytes) 000268 1763876605.637970 MIRROR_RESYNC 5 mirrors 000269 1763876605.921109 MIRROR_EXTEND to 6 mirrors 000270 1763876606.093769 trunc from 0x031487 to 0x03276e (0x12e8 bytes) 000271 1763876606.109576 read 0x01d8a5 thru 0x01db75 (0x2d1 bytes) 000273 1763876606.111485 mapwrite 0x037515 thru 0x03ffff (0x8aeb bytes) 000274 1763876606.124265 trunc from 0x01fb30 to 0x021ccd (0x219e bytes) 000275 1763876606.125920 write 0x03faf6 thru 0x03ffff (0x50a bytes) 000276 1763876606.126475 trunc from 0x040000 to 0x02c432 (0x13bcd bytes) 000277 1763876606.131134 punch from 0x016be4 to 0x019b94 (0x2fb1 bytes) 000278 1763876606.132832 read 0x01b3c8 thru 0x029863 (0xe49c bytes) 000279 1763876606.135374 punch from 0x01ea63 to 0x028e7a (0xa418 bytes) 000281 1763876606.139006 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2c433): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c433): Mirror 4: 0x58201e92 (stale) Mirror 5: 0x58201e92 (stale) Mirror 6: 0x58201e92 (stale) Mirror 7: 0x58201e92 (stale) Mirror 8: 0x58201e92 (stale) CRC-32 checksum value for chunk [0, 0x2c433): Mirror 1: 0xbafa462c Mirror 9: 0xbafa462c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c433: skipped 000282 1763876606.383585 fallocate 0x0009e4 thru 0x006fd3 (0x65f0 bytes) 000283 1763876606.393584 mapwrite 0x00095f thru 0x003ac3 (0x3165 bytes) 000284 1763876606.400720 fallocate 0x00d227 thru 0x00ede1 (0x1bbb bytes) 000285 1763876606.402358 write 0x018a24 thru 0x0232fa (0xa8d7 bytes) 000287 1763876606.404712 read 0x01894d thru 0x01b8d0 (0x2f84 bytes) 000288 1763876606.405078 punch from 0x0115cf to 0x0167e9 (0x521b bytes) 000289 1763876606.406832 trunc from 0x00746f to 0x00c7df (0x5371 bytes) 000290 1763876606.408726 fallocate 0x02d909 thru 0x03a570 (0xcc68 bytes) 000291 1763876606.410791 mapread 0x02ba43 thru 0x02c432 (0x9f0 bytes) 000292 1763876606.412815 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c433): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c433): Mirror 4: 0x58201e92 (stale) Mirror 5: 0x58201e92 (stale) Mirror 6: 0x58201e92 (stale) Mirror 7: 0x58201e92 (stale) Mirror 8: 0x58201e92 (stale) Mirror 9: 0xbafa462c (stale) CRC-32 checksum value for chunk [0, 0x2c433): Mirror 1: 0x19d30521 Mirror 10: 0x19d30521 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c433: skipped 000293 1763876606.654825 trunc from 0x02c433 to 0x03fa47 (0x13615 bytes) 000294 1763876606.671168 read 0x039c98 thru 0x03fa47 (0x5db0 bytes) 000295 1763876606.673881 read 0x03d77f thru 0x03fa47 (0x22c9 bytes) 000296 1763876606.674229 trunc from 0x03fa48 to 0x023019 (0x1ca2e bytes) 000297 1763876606.679905 mapread 0x0216c2 thru 0x023019 (0x1958 bytes) 000298 1763876606.683087 trunc from 0x00202d to 0x00d85a (0xb82e bytes) 000299 1763876606.684812 trunc from 0x013272 to 0x018972 (0x5701 bytes) 000300 1763876606.686410 read 0x02235b thru 0x023019 (0xcbf bytes) 000301 1763876606.686907 write 0x00f8e4 thru 0x00f9ad (0xca bytes) 000302 1763876606.687529 trunc from 0x00257f to 0x00714a (0x4bcc bytes) 000303 1763876606.689299 mapwrite 0x02b00d thru 0x03ac6e (0xfc62 bytes) 000304 1763876606.704335 fallocate 0x00d76f thru 0x011a86 (0x4318 bytes) 000305 1763876606.706131 mapread 0x023ddf thru 0x02dba7 (0x9dc9 bytes) 000306 1763876606.707011 trunc from 0x038d91 to 0x03ac6e (0x1ede bytes) 000307 1763876606.708937 write 0x01a0c4 thru 0x01c6ce (0x260b bytes) 000308 1763876606.709847 fallocate 0x001dde thru 0x0114a7 (0xf6ca bytes) 000309 1763876606.712839 MIRROR_RESYNC 8 mirrors 000310 1763876607.203782 trunc from 0x03ac6f to 0x004106 (0x36b68 bytes) 000311 1763876607.214983 punch from 0x001666 to 0x002d3a (0x16d5 bytes) 000312 1763876607.216456 punch from 0x0027fa to 0x004106 (0x190d bytes) 000313 1763876607.217805 trunc from 0x004107 to 0x01436c (0x10266 bytes) 000314 1763876607.223165 trunc from 0x01436d to 0x025739 (0x113cd bytes) 000316 1763876607.228395 trunc from 0x02573a to 0x018763 (0xcfd6 bytes) 000317 1763876607.232985 read 0x003834 thru 0x00ab13 (0x72e0 bytes) 000318 1763876607.235592 mapread 0x00241d thru 0x00c4b1 (0xa095 bytes) 000319 1763876607.240648 MIRROR_RESYNC 8 mirrors 000320 1763876607.518229 MIRROR_EXTEND to 9 mirrors 000321 1763876607.707894 trunc from 0x015e8b to 0x018763 (0x28d9 bytes) 000322 1763876607.720067 trunc from 0x018764 to 0x006307 (0x1245c bytes) 000323 1763876607.729944 trunc from 0x004036 to 0x006307 (0x22d2 bytes) 000324 1763876607.731573 mapread 0x0005fd thru 0x006307 (0x5d0b bytes) 000325 1763876607.736035 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6308): Mirror 4: 0x4fba8110 (stale) Mirror 5: 0x4fba8110 (stale) Mirror 6: 0x4fba8110 (stale) Mirror 8: 0x4fba8110 (stale) Mirror 9: 0x4fba8110 (stale) Mirror 10: 0x4fba8110 (stale) Mirror 11: 0x4fba8110 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6308): Mirror 4: 0x4fba8110 (stale) Mirror 5: 0x4fba8110 (stale) Mirror 6: 0x4fba8110 (stale) Mirror 8: 0x4fba8110 (stale) Mirror 9: 0x4fba8110 (stale) Mirror 10: 0x4fba8110 (stale) Mirror 11: 0x4fba8110 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6308: skipped 000326 1763876607.838911 read 0x0027d9 thru 0x006307 (0x3b2f bytes) 000328 1763876607.842582 trunc from 0x006050 to 0x006307 (0x2b8 bytes) 000330 1763876607.844583 mapread 0x003b7a thru 0x006307 (0x278e bytes) 000331 1763876607.847205 fallocate 0x02bc6a thru 0x02f22d (0x35c4 bytes) 000332 1763876607.849189 fallocate 0x03fd7c thru 0x03ffff (0x284 bytes) 000333 1763876607.851307 mapwrite 0x0262bf thru 0x02f91c (0x965e bytes) 000334 1763876607.863553 MIRROR_RESYNC 8 mirrors 000335 1763876608.219424 punch from 0x00bd37 to 0x01ae14 (0xf0de bytes) 000336 1763876608.226758 mapwrite 0x010982 thru 0x0127f9 (0x1e78 bytes) 000337 1763876608.234321 read 0x00a5cc thru 0x01557c (0xafb1 bytes) 000338 1763876608.237129 mapread 0x03dccf thru 0x03ffff (0x2331 bytes) 000339 1763876608.237756 fallocate 0x012ac4 thru 0x01a0b4 (0x75f1 bytes) 000340 1763876608.239496 trunc from 0x020f6d to 0x02fcf3 (0xed87 bytes) 000341 1763876608.241094 mapwrite 0x03fc7b thru 0x03ffff (0x385 bytes) 000343 1763876608.245174 fallocate 0x018170 thru 0x024fc5 (0xce56 bytes) 000344 1763876608.248655 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2db643ec (stale) Mirror 5: 0x2db643ec (stale) Mirror 6: 0x2db643ec (stale) Mirror 8: 0x2db643ec (stale) Mirror 9: 0x2db643ec (stale) Mirror 10: 0x2db643ec (stale) Mirror 11: 0x2db643ec (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1c872400 Mirror 12: 0x1c872400 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000345 1763876608.490769 punch from 0x039f16 to 0x03d20c (0x32f7 bytes) 000346 1763876608.499725 read 0x029cdf thru 0x02a433 (0x755 bytes) 000347 1763876608.501540 punch from 0x00b817 to 0x00f778 (0x3f62 bytes) 000348 1763876608.503077 trunc from 0x040000 to 0x02f133 (0x10ecc bytes) 000349 1763876608.513052 trunc from 0x02f134 to 0x021dd5 (0xd35e bytes) 000350 1763876608.517668 MIRROR_RESYNC 9 mirrors 000351 1763876608.848975 MIRROR_EXTEND to 10 mirrors 000352 1763876609.021585 trunc from 0x0086e4 to 0x01261a (0x9f37 bytes) 000353 1763876609.032875 write 0x0328da thru 0x03e6ab (0xbdd2 bytes) 000354 1763876609.035316 mapread 0x01d05c thru 0x02c168 (0xf10d bytes) 000355 1763876609.040374 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e6ac): Mirror 4: 0x4c5a86bd (stale) Mirror 5: 0x4c5a86bd (stale) Mirror 6: 0x4c5a86bd (stale) Mirror 8: 0x4c5a86bd (stale) Mirror 9: 0x4c5a86bd (stale) Mirror 10: 0x4c5a86bd (stale) Mirror 12: 0x4c5a86bd (stale) Mirror 13: 0x4c5a86bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6ac): Mirror 4: 0x4c5a86bd (stale) Mirror 5: 0x4c5a86bd (stale) Mirror 6: 0x4c5a86bd (stale) Mirror 8: 0x4c5a86bd (stale) Mirror 9: 0x4c5a86bd (stale) Mirror 10: 0x4c5a86bd (stale) Mirror 12: 0x4c5a86bd (stale) Mirror 13: 0x4c5a86bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6ac: skipped 000356 1763876609.175058 fallocate 0x03572c thru 0x03a7b8 (0x508d bytes) 000357 1763876609.176786 fallocate 0x0134dc thru 0x01fc57 (0xc77c bytes) 000358 1763876609.178415 mapwrite 0x0383ae thru 0x0391d7 (0xe2a bytes) 000359 1763876609.183854 trunc from 0x0204e5 to 0x02f560 (0xf07c bytes) 000360 1763876609.185347 fallocate 0x0200cf thru 0x02294c (0x287e bytes) 000362 1763876609.187106 punch from 0x007a1e to 0x014075 (0xc658 bytes) 000363 1763876609.188748 read 0x02ed3a thru 0x031220 (0x24e7 bytes) 000364 1763876609.192837 MIRROR_RESYNC 9 mirrors 000365 1763876609.478870 MIRROR_RESYNC 9 mirrors 000366 1763876609.626424 mapread 0x021e63 thru 0x02b8a6 (0x9a44 bytes) 000367 1763876609.629664 mapwrite 0x01121a thru 0x018e48 (0x7c2f bytes) 000368 1763876609.668030 write 0x002170 thru 0x00f823 (0xd6b4 bytes) 000369 1763876609.668903 trunc from 0x03e6ac to 0x033747 (0xaf64 bytes) 000371 1763876609.678237 fallocate 0x03afae thru 0x03ffff (0x5052 bytes) 000372 1763876609.679953 fallocate 0x031716 thru 0x033bd1 (0x24bc bytes) 000373 1763876609.681748 trunc from 0x0170c5 to 0x0236ee (0xc62a bytes) 000374 1763876609.683142 mapwrite 0x02b410 thru 0x038126 (0xcd17 bytes) 000375 1763876609.695398 write 0x03e196 thru 0x03ea56 (0x8c1 bytes) 000376 1763876609.696049 mapwrite 0x0118a7 thru 0x0194c7 (0x7c21 bytes) 000377 1763876609.701270 fallocate 0x02e5ce thru 0x03db09 (0xf53c bytes) 000378 1763876609.702758 trunc from 0x03ea57 to 0x038542 (0x6514 bytes) 000379 1763876609.706619 punch from 0x028503 to 0x03705f (0xeb5d bytes) 000380 1763876609.708207 trunc from 0x038543 to 0x0264e1 (0x12061 bytes) 000381 1763876609.712385 trunc from 0x0264e2 to 0x0340d2 (0xdbf1 bytes) 000382 1763876609.715914 trunc from 0x00c76a to 0x010cfa (0x4591 bytes) 000383 1763876609.717555 trunc from 0x0275b3 to 0x02b32a (0x3d78 bytes) 000384 1763876609.719093 trunc from 0x0340d3 to 0x02956f (0xab63 bytes) 000385 1763876609.723480 punch from 0x00afd7 to 0x010f17 (0x5f41 bytes) 000386 1763876609.725333 trunc from 0x006409 to 0x01591b (0xf513 bytes) 000387 1763876609.727841 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29570): Mirror 5: 0xc7aa82a7 (stale) Mirror 6: 0xc7aa82a7 (stale) Mirror 8: 0xc7aa82a7 (stale) Mirror 9: 0xc7aa82a7 (stale) Mirror 10: 0xc7aa82a7 (stale) Mirror 12: 0xc7aa82a7 (stale) Mirror 13: 0xc7aa82a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29570): Mirror 5: 0xc7aa82a7 (stale) Mirror 6: 0xc7aa82a7 (stale) Mirror 8: 0xc7aa82a7 (stale) Mirror 9: 0xc7aa82a7 (stale) Mirror 10: 0xc7aa82a7 (stale) Mirror 12: 0xc7aa82a7 (stale) Mirror 13: 0xc7aa82a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29570: skipped 000388 1763876609.833001 punch from 0x007aef to 0x011114 (0x9626 bytes) 000390 1763876609.834937 fallocate 0x0391f7 thru 0x03a62d (0x1437 bytes) 000391 1763876609.836511 write 0x01cb5a thru 0x01e85c (0x1d03 bytes) 000392 1763876609.840930 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3a62e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a62e): Mirror 5: 0x10dd1335 (stale) Mirror 6: 0x10dd1335 (stale) Mirror 8: 0x10dd1335 (stale) Mirror 9: 0x10dd1335 (stale) Mirror 10: 0x10dd1335 (stale) Mirror 12: 0x10dd1335 (stale) Mirror 13: 0x10dd1335 (stale) CRC-32 checksum value for chunk [0, 0x3a62e): Mirror 1: 0x21f895e0 Mirror 14: 0x21f895e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a62e: skipped 000395 1763876610.082768 mapwrite 0x00d97f thru 0x01c8e5 (0xef67 bytes) 000396 1763876610.120678 mapwrite 0x030882 thru 0x034317 (0x3a96 bytes) 000397 1763876610.126268 fallocate 0x03b489 thru 0x03ffff (0x4b77 bytes) 000398 1763876610.128047 mapwrite 0x026be2 thru 0x035acc (0xeeeb bytes) 000399 1763876610.135737 write 0x03728b thru 0x03ca86 (0x57fc bytes) 000400 1763876610.137781 mapread 0x02b64a thru 0x03310a (0x7ac1 bytes) 000401 1763876610.138373 mapwrite 0x00a161 thru 0x014cd5 (0xab75 bytes) 000402 1763876610.144480 read 0x03ae0e thru 0x03ca86 (0x1c79 bytes) 000403 1763876610.144818 mapread 0x0246b1 thru 0x024bda (0x52a bytes) 000405 1763876610.145591 trunc from 0x03ca87 to 0x0256ed (0x17399 bytes) 000406 1763876610.154798 trunc from 0x004465 to 0x008cc4 (0x4860 bytes) 000408 1763876610.156874 read 0x0129bc thru 0x01605a (0x369f bytes) 000409 1763876610.158807 MIRROR_RESYNC 9 mirrors 000410 1763876610.498468 punch from 0x0218d3 to 0x0256ed (0x3e1b bytes) 000412 1763876610.511141 read 0x004917 thru 0x0132fd (0xe9e7 bytes) 000413 1763876610.514165 mapwrite 0x0340c0 thru 0x03a47b (0x63bc bytes) 000415 1763876610.534609 write 0x00bbe9 thru 0x00e17c (0x2594 bytes) 000417 1763876610.535681 write 0x0250fd thru 0x03149a (0xc39e bytes) 000419 1763876610.536757 write 0x02e5f8 thru 0x036e51 (0x885a bytes) 000420 1763876610.537515 write 0x00a65d thru 0x00f2b6 (0x4c5a bytes) 000421 1763876610.538156 trunc from 0x03a47c to 0x01e015 (0x1c466 bytes) 000422 1763876610.543089 mapwrite 0x001e72 thru 0x003953 (0x1ae2 bytes) 000424 1763876610.550077 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1e016): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e016): Mirror 5: 0x6bc39933 (stale) Mirror 6: 0x6bc39933 (stale) Mirror 8: 0x6bc39933 (stale) Mirror 9: 0x6bc39933 (stale) Mirror 10: 0x6bc39933 (stale) Mirror 12: 0x6bc39933 (stale) Mirror 13: 0x6bc39933 (stale) Mirror 14: 0x6bc39933 (stale) CRC-32 checksum value for chunk [0, 0x1e016): Mirror 1: 0xb73aeb4f Mirror 15: 0xb73aeb4f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e016: skipped 000425 1763876610.772698 fallocate 0x0236cd thru 0x0246c3 (0xff7 bytes) 000426 1763876610.785209 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x246c4): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246c4): Mirror 5: 0x8d32a5f8 (stale) Mirror 6: 0x8d32a5f8 (stale) Mirror 8: 0x8d32a5f8 (stale) Mirror 9: 0x8d32a5f8 (stale) Mirror 10: 0x8d32a5f8 (stale) Mirror 12: 0x8d32a5f8 (stale) Mirror 13: 0x8d32a5f8 (stale) Mirror 14: 0x8d32a5f8 (stale) Mirror 15: 0xa6dd7547 (stale) CRC-32 checksum value for chunk [0, 0x246c4): Mirror 1: 0xa6dd7547 Mirror 16: 0xa6dd7547 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246c4: skipped 000427 1763876611.022681 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x246c4): Mirror 5: 0x8d32a5f8 (stale) Mirror 6: 0x8d32a5f8 (stale) Mirror 9: 0x8d32a5f8 (stale) Mirror 10: 0x8d32a5f8 (stale) Mirror 12: 0x8d32a5f8 (stale) Mirror 13: 0x8d32a5f8 (stale) Mirror 14: 0x8d32a5f8 (stale) Mirror 15: 0xa6dd7547 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246c4): Mirror 5: 0x8d32a5f8 (stale) Mirror 6: 0x8d32a5f8 (stale) Mirror 9: 0x8d32a5f8 (stale) Mirror 10: 0x8d32a5f8 (stale) Mirror 12: 0x8d32a5f8 (stale) Mirror 13: 0x8d32a5f8 (stale) Mirror 14: 0x8d32a5f8 (stale) Mirror 15: 0xa6dd7547 (stale) CRC-32 checksum value for chunk [0, 0x246c4): Mirror 1: 0xa6dd7547 Mirror 16: 0xa6dd7547 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246c4: skipped 000428 1763876611.249671 MIRROR_RESYNC 10 mirrors 000429 1763876611.617713 punch from 0x000c6b to 0x00aa57 (0x9ded bytes) 000432 1763876611.627313 punch from 0x019be6 to 0x021e20 (0x823b bytes) 000433 1763876611.629028 write 0x023b71 thru 0x02dc9a (0xa12a bytes) 000434 1763876611.630978 read 0x0127a9 thru 0x01bb9d (0x93f5 bytes) 000435 1763876611.633381 trunc from 0x02dc9b to 0x036640 (0x89a6 bytes) 000436 1763876611.643184 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36641): Mirror 5: 0xf58aeb7 (stale) Mirror 6: 0xf58aeb7 (stale) Mirror 9: 0xf58aeb7 (stale) Mirror 10: 0xf58aeb7 (stale) Mirror 12: 0xf58aeb7 (stale) Mirror 13: 0xf58aeb7 (stale) Mirror 14: 0xf58aeb7 (stale) Mirror 15: 0xf58aeb7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36641): Mirror 5: 0xf58aeb7 (stale) Mirror 6: 0xf58aeb7 (stale) Mirror 9: 0xf58aeb7 (stale) Mirror 10: 0xf58aeb7 (stale) Mirror 12: 0xf58aeb7 (stale) Mirror 13: 0xf58aeb7 (stale) Mirror 14: 0xf58aeb7 (stale) Mirror 15: 0xf58aeb7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36641: skipped 000437 1763876611.764136 trunc from 0x036641 to 0x01bd86 (0x1a8ba bytes) 000439 1763876611.769905 trunc from 0x01bd87 to 0x03b6f4 (0x1f96e bytes) 000440 1763876611.775968 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b6f5): Mirror 5: 0x24afcea5 (stale) Mirror 6: 0x24afcea5 (stale) Mirror 9: 0x24afcea5 (stale) Mirror 10: 0x24afcea5 (stale) Mirror 12: 0x24afcea5 (stale) Mirror 13: 0x24afcea5 (stale) Mirror 14: 0x24afcea5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6f5): Mirror 5: 0x24afcea5 (stale) Mirror 6: 0x24afcea5 (stale) Mirror 9: 0x24afcea5 (stale) Mirror 10: 0x24afcea5 (stale) Mirror 12: 0x24afcea5 (stale) Mirror 13: 0x24afcea5 (stale) Mirror 14: 0x24afcea5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6f5: skipped 000441 1763876611.882953 trunc from 0x01bf62 to 0x01f657 (0x36f6 bytes) 000442 1763876611.884636 punch from 0x02b5bc to 0x02ed94 (0x37d9 bytes) 000443 1763876611.885897 punch from 0x02a224 to 0x030b6e (0x694b bytes) 000444 1763876611.887293 trunc from 0x03b6f5 to 0x0100a0 (0x2b654 bytes) 000445 1763876611.892120 mapwrite 0x038ded thru 0x03ffff (0x7213 bytes) 000446 1763876611.903851 fallocate 0x004bd6 thru 0x00bf49 (0x7374 bytes) 000447 1763876611.905427 write 0x009467 thru 0x00ceb6 (0x3a50 bytes) 000448 1763876611.908013 trunc from 0x040000 to 0x034a6f (0xb590 bytes) 000449 1763876611.913018 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x34a70): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a70): Mirror 5: 0x719f757e (stale) Mirror 6: 0x719f757e (stale) Mirror 9: 0x719f757e (stale) Mirror 10: 0x719f757e (stale) Mirror 12: 0x719f757e (stale) Mirror 13: 0x719f757e (stale) Mirror 14: 0x719f757e (stale) CRC-32 checksum value for chunk [0, 0x34a70): Mirror 1: 0x9d0d629c Mirror 15: 0x9d0d629c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a70: skipped 000451 1763876612.159208 read 0x0268ac thru 0x028928 (0x207d bytes) 000452 1763876612.163857 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x34a70): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a70): Mirror 5: 0x719f757e (stale) Mirror 6: 0x719f757e (stale) Mirror 9: 0x719f757e (stale) Mirror 10: 0x719f757e (stale) Mirror 12: 0x719f757e (stale) Mirror 13: 0x719f757e (stale) Mirror 14: 0x719f757e (stale) CRC-32 checksum value for chunk [0, 0x34a70): Mirror 1: 0x9d0d629c Mirror 15: 0x9d0d629c Mirror 16: 0x9d0d629c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a70: skipped 000453 1763876612.386401 trunc from 0x034a70 to 0x039d6a (0x52fb bytes) 000455 1763876612.398443 mapwrite 0x00958f thru 0x0139ec (0xa45e bytes) 000456 1763876612.406283 trunc from 0x039d6b to 0x01f2ab (0x1aabf bytes) 000457 1763876612.410697 fallocate 0x014224 thru 0x023156 (0xef33 bytes) 000458 1763876612.412216 mapread 0x013425 thru 0x01b22c (0x7e08 bytes) 000459 1763876612.412752 fallocate 0x01c4c9 thru 0x0294c2 (0xcffa bytes) 000461 1763876612.414312 mapwrite 0x00010e thru 0x00743b (0x732e bytes) 000462 1763876612.421919 read 0x012887 thru 0x0171fc (0x4976 bytes) 000463 1763876612.422586 read 0x009e46 thru 0x00d921 (0x3adc bytes) 000464 1763876612.423097 mapwrite 0x013317 thru 0x01700a (0x3cf4 bytes) 000465 1763876612.428063 mapwrite 0x006a4e thru 0x01498e (0xdf41 bytes) 000467 1763876612.435154 trunc from 0x0294c3 to 0x01ff70 (0x9552 bytes) 000469 1763876612.440417 mapwrite 0x000cc4 thru 0x00eb67 (0xdea4 bytes) 000470 1763876612.448092 write 0x026836 thru 0x02dd0e (0x74d9 bytes) 000471 1763876612.448852 mapwrite 0x00c557 thru 0x01a120 (0xdbca bytes) 000472 1763876612.455984 trunc from 0x024fa1 to 0x02dd0e (0x8d6e bytes) 000474 1763876612.459558 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2dd0f): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd0f): Mirror 5: 0xd963588b (stale) Mirror 6: 0xd963588b (stale) Mirror 9: 0xd963588b (stale) Mirror 10: 0xd963588b (stale) Mirror 12: 0xd963588b (stale) Mirror 13: 0xd963588b (stale) Mirror 14: 0xd963588b (stale) Mirror 15: 0xe2267c22 (stale) Mirror 16: 0xe2267c22 (stale) CRC-32 checksum value for chunk [0, 0x2dd0f): Mirror 1: 0xb876aadb Mirror 17: 0xb876aadb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd0f: skipped 000475 1763876612.683021 write 0x0109e0 thru 0x018c9b (0x82bc bytes) 000477 1763876612.695357 trunc from 0x01f6ac to 0x01f8fd (0x252 bytes) 000478 1763876612.696708 punch from 0x010c29 to 0x015b80 (0x4f58 bytes) 000479 1763876612.699856 mapread 0x016e11 thru 0x023fbe (0xd1ae bytes) 000480 1763876612.702506 trunc from 0x02dd0f to 0x00b446 (0x228c8 bytes) 000481 1763876612.711130 write 0x00ad8e thru 0x01a0ea (0xf35d bytes) 000482 1763876612.714076 trunc from 0x003305 to 0x00e1bd (0xaeb9 bytes) 000483 1763876612.718210 fallocate 0x021294 thru 0x027a3d (0x67aa bytes) 000484 1763876612.719834 write 0x0046ec thru 0x011aab (0xd3c0 bytes) 000485 1763876612.723003 mapread 0x010020 thru 0x019b23 (0x9b04 bytes) 000486 1763876612.723643 read 0x02644a thru 0x02734b (0xf02 bytes) 000487 1763876612.725983 fallocate 0x005f1a thru 0x01099d (0xaa84 bytes) 000488 1763876612.727907 punch from 0x018e47 to 0x01fb6f (0x6d29 bytes) 000489 1763876612.731281 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x27a3e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a3e): Mirror 5: 0x63b42173 (stale) Mirror 6: 0x63b42173 (stale) Mirror 9: 0x63b42173 (stale) Mirror 10: 0x63b42173 (stale) Mirror 12: 0x63b42173 (stale) Mirror 13: 0x63b42173 (stale) Mirror 14: 0x63b42173 (stale) Mirror 15: 0xfd0c0bd3 (stale) Mirror 16: 0xfd0c0bd3 (stale) Mirror 17: 0x92114e9e (stale) CRC-32 checksum value for chunk [0, 0x27a3e): Mirror 1: 0x9240daba Mirror 18: 0x9240daba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a3e: skipped 000491 1763876612.955605 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x27a3e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a3e): Mirror 5: 0x63b42173 (stale) Mirror 6: 0x63b42173 (stale) Mirror 9: 0x63b42173 (stale) Mirror 10: 0x63b42173 (stale) Mirror 12: 0x63b42173 (stale) Mirror 13: 0x63b42173 (stale) Mirror 14: 0x63b42173 (stale) Mirror 15: 0xfd0c0bd3 (stale) Mirror 16: 0xfd0c0bd3 (stale) Mirror 17: 0x92114e9e (stale) CRC-32 checksum value for chunk [0, 0x27a3e): Mirror 1: 0x9240daba Mirror 18: 0x9240daba Mirror 19: 0x9240daba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a3e: skipped 000492 1763876613.192568 MIRROR_RESYNC 13 mirrors 000493 1763876613.525045 fallocate 0x02e05f thru 0x03b07e (0xd020 bytes) 000494 1763876613.538591 punch from 0x011317 to 0x017d15 (0x69ff bytes) 000495 1763876613.540223 punch from 0x016750 to 0x01edfd (0x86ae bytes) 000496 1763876613.541911 trunc from 0x018b4f to 0x01c918 (0x3dca bytes) 000497 1763876613.543451 read 0x002bce thru 0x00c5ca (0x99fd bytes) 000498 1763876613.545998 trunc from 0x027a3e to 0x0355ff (0xdbc2 bytes) 000499 1763876613.555612 mapwrite 0x013aa7 thru 0x014504 (0xa5e bytes) 000500 1763876613.562609 mapread 0x00c135 thru 0x00c27c (0x148 bytes) 000501 1763876613.563192 trunc from 0x011e4e to 0x013aa3 (0x1c56 bytes) 000502 1763876613.564801 punch from 0x00c48b to 0x01bdb5 (0xf92b bytes) 000503 1763876613.566404 trunc from 0x0191fd to 0x01fe98 (0x6c9c bytes) 000504 1763876613.567786 punch from 0x01ba1c to 0x01d2f5 (0x18da bytes) 000505 1763876613.569012 mapread 0x01662e thru 0x022626 (0xbff9 bytes) 000506 1763876613.572080 write 0x01ef20 thru 0x026e9b (0x7f7c bytes) 000508 1763876613.573459 mapread 0x010b04 thru 0x01bce9 (0xb1e6 bytes) 000509 1763876613.574181 trunc from 0x035600 to 0x0140bd (0x21542 bytes) 000510 1763876613.580459 MIRROR_RESYNC 13 mirrors 000511 1763876613.912709 punch from 0x012120 to 0x0140bd (0x1f9e bytes) 000512 1763876613.921665 write 0x02a9aa thru 0x02be7c (0x14d3 bytes) 000513 1763876613.922621 trunc from 0x02be7d to 0x00649b (0x259e1 bytes) 000514 1763876613.931689 trunc from 0x00124c to 0x004cbd (0x3a72 bytes) 000515 1763876613.933197 punch from 0x005792 to 0x00649b (0xd0a bytes) 000516 1763876613.934453 read 0x006394 thru 0x00649b (0x108 bytes) 000517 1763876613.935869 write 0x0181e6 thru 0x01f0af (0x6eca bytes) 000518 1763876613.936624 write 0x029fa7 thru 0x0330e9 (0x9143 bytes) 000519 1763876613.938877 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x330ea): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x330ea): Mirror 5: 0xbb6679f0 (stale) Mirror 6: 0xbb6679f0 (stale) Mirror 9: 0xbb6679f0 (stale) Mirror 10: 0xbb6679f0 (stale) Mirror 12: 0xbb6679f0 (stale) Mirror 13: 0xbb6679f0 (stale) Mirror 14: 0xbb6679f0 (stale) Mirror 15: 0xbb6679f0 (stale) Mirror 16: 0xbb6679f0 (stale) Mirror 17: 0xbb6679f0 (stale) Mirror 18: 0xbb6679f0 (stale) Mirror 19: 0xbb6679f0 (stale) CRC-32 checksum value for chunk [0, 0x330ea): Mirror 1: 0x84fc28b3 Mirror 20: 0x84fc28b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x330ea: skipped 000521 1763876614.222273 fallocate 0x023241 thru 0x024b95 (0x1955 bytes) 000522 1763876614.233574 mapwrite 0x0175b1 thru 0x0260b4 (0xeb04 bytes) 000523 1763876614.245291 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x330ea): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x330ea): Mirror 5: 0xbb6679f0 (stale) Mirror 6: 0xbb6679f0 (stale) Mirror 9: 0xbb6679f0 (stale) Mirror 10: 0xbb6679f0 (stale) Mirror 12: 0xbb6679f0 (stale) Mirror 13: 0xbb6679f0 (stale) Mirror 14: 0xbb6679f0 (stale) Mirror 15: 0xbb6679f0 (stale) Mirror 16: 0xbb6679f0 (stale) Mirror 17: 0xbb6679f0 (stale) Mirror 18: 0xbb6679f0 (stale) Mirror 19: 0xbb6679f0 (stale) Mirror 20: 0x84fc28b3 (stale) CRC-32 checksum value for chunk [0, 0x330ea): Mirror 1: 0x2ed0508b Mirror 21: 0x2ed0508b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x330ea: skipped 000524 1763876614.514420 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x330ea): Mirror 6: 0xbb6679f0 (stale) Mirror 9: 0xbb6679f0 (stale) Mirror 10: 0xbb6679f0 (stale) Mirror 12: 0xbb6679f0 (stale) Mirror 13: 0xbb6679f0 (stale) Mirror 14: 0xbb6679f0 (stale) Mirror 15: 0xbb6679f0 (stale) Mirror 16: 0xbb6679f0 (stale) Mirror 17: 0xbb6679f0 (stale) Mirror 18: 0xbb6679f0 (stale) Mirror 19: 0xbb6679f0 (stale) Mirror 20: 0x84fc28b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x330ea): Mirror 6: 0xbb6679f0 (stale) Mirror 9: 0xbb6679f0 (stale) Mirror 10: 0xbb6679f0 (stale) Mirror 12: 0xbb6679f0 (stale) Mirror 13: 0xbb6679f0 (stale) Mirror 14: 0xbb6679f0 (stale) Mirror 15: 0xbb6679f0 (stale) Mirror 16: 0xbb6679f0 (stale) Mirror 17: 0xbb6679f0 (stale) Mirror 18: 0xbb6679f0 (stale) Mirror 19: 0xbb6679f0 (stale) Mirror 20: 0x84fc28b3 (stale) CRC-32 checksum value for chunk [0, 0x330ea): Mirror 1: 0x2ed0508b Mirror 21: 0x2ed0508b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x330ea: skipped 000525 1763876614.805607 trunc from 0x0330ea to 0x018539 (0x1abb0 bytes) 000527 1763876614.823072 trunc from 0x00a40b to 0x013796 (0x938c bytes) 000528 1763876614.824618 write 0x01711d thru 0x022a8d (0xb971 bytes) 000530 1763876614.827017 mapread 0x01efd2 thru 0x022a8d (0x3abc bytes) 000532 1763876614.827840 fallocate 0x022214 thru 0x024d8d (0x2b7a bytes) 000533 1763876614.829528 mapwrite 0x036111 thru 0x03ffff (0x9eef bytes) 000534 1763876614.841923 write 0x02719b thru 0x02fc4d (0x8ab3 bytes) 000535 1763876614.846217 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1dfebd63 (stale) Mirror 9: 0x1dfebd63 (stale) Mirror 10: 0x1dfebd63 (stale) Mirror 12: 0x1dfebd63 (stale) Mirror 13: 0x1dfebd63 (stale) Mirror 14: 0x1dfebd63 (stale) Mirror 15: 0x1dfebd63 (stale) Mirror 16: 0x1dfebd63 (stale) Mirror 17: 0x1dfebd63 (stale) Mirror 18: 0x1dfebd63 (stale) Mirror 19: 0x1dfebd63 (stale) Mirror 21: 0x38b2e8c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1dfebd63 (stale) Mirror 9: 0x1dfebd63 (stale) Mirror 10: 0x1dfebd63 (stale) Mirror 12: 0x1dfebd63 (stale) Mirror 13: 0x1dfebd63 (stale) Mirror 14: 0x1dfebd63 (stale) Mirror 15: 0x1dfebd63 (stale) Mirror 16: 0x1dfebd63 (stale) Mirror 17: 0x1dfebd63 (stale) Mirror 18: 0x1dfebd63 (stale) Mirror 19: 0x1dfebd63 (stale) Mirror 21: 0x38b2e8c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000536 1763876614.997997 mapwrite 0x00ec9e thru 0x015d5d (0x70c0 bytes) 000537 1763876615.007275 write 0x0252f4 thru 0x033223 (0xdf30 bytes) 000538 1763876615.009575 trunc from 0x040000 to 0x0232fd (0x1cd02 bytes) 000539 1763876615.014182 fallocate 0x00bef8 thru 0x00da0c (0x1b15 bytes) 000540 1763876615.015690 trunc from 0x0232fe to 0x02af66 (0x7c69 bytes) 000541 1763876615.019413 trunc from 0x010d1b to 0x01baea (0xadd0 bytes) 000542 1763876615.022554 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2af67): Mirror 6: 0x6a7c273f (stale) Mirror 10: 0x6a7c273f (stale) Mirror 12: 0x6a7c273f (stale) Mirror 13: 0x6a7c273f (stale) Mirror 14: 0x6a7c273f (stale) Mirror 15: 0x6a7c273f (stale) Mirror 16: 0x6a7c273f (stale) Mirror 17: 0x6a7c273f (stale) Mirror 18: 0x6a7c273f (stale) Mirror 19: 0x6a7c273f (stale) Mirror 21: 0xf7e1dae8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2af67): Mirror 6: 0x6a7c273f (stale) Mirror 10: 0x6a7c273f (stale) Mirror 12: 0x6a7c273f (stale) Mirror 13: 0x6a7c273f (stale) Mirror 14: 0x6a7c273f (stale) Mirror 15: 0x6a7c273f (stale) Mirror 16: 0x6a7c273f (stale) Mirror 17: 0x6a7c273f (stale) Mirror 18: 0x6a7c273f (stale) Mirror 19: 0x6a7c273f (stale) Mirror 21: 0xf7e1dae8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2af67: skipped 000543 1763876615.163261 MIRROR_RESYNC 12 mirrors 000544 1763876615.783889 read 0x003d08 thru 0x00c046 (0x833f bytes) 000545 1763876615.786477 trunc from 0x00fbd2 to 0x0102d4 (0x703 bytes) 000547 1763876615.798707 MIRROR_RESYNC 12 mirrors 000548 1763876616.405790 mapread 0x01d079 thru 0x01eed7 (0x1e5f bytes) 000550 1763876616.408886 trunc from 0x018cfe to 0x019289 (0x58c bytes) 000551 1763876616.417909 trunc from 0x012dc7 to 0x014f10 (0x214a bytes) 000553 1763876616.419833 fallocate 0x013f58 thru 0x014c3b (0xce4 bytes) 000554 1763876616.422792 MIRROR_RESYNC 12 mirrors 000555 1763876617.013656 trunc from 0x013566 to 0x021c88 (0xe723 bytes) 000557 1763876617.023624 trunc from 0x02af67 to 0x007bb9 (0x233ad bytes) 000558 1763876617.028613 write 0x019a9d thru 0x021c83 (0x81e7 bytes) 000559 1763876617.029542 mapread 0x009b71 thru 0x01579a (0xbc2a bytes) 000560 1763876617.032457 mapread 0x010031 thru 0x019254 (0x9224 bytes) 000561 1763876617.033328 mapread 0x01f2ac thru 0x021c83 (0x29d8 bytes) 000562 1763876617.034031 write 0x039303 thru 0x03ffff (0x6cfd bytes) 000563 1763876617.035144 trunc from 0x001530 to 0x0099d5 (0x84a6 bytes) 000564 1763876617.036894 read 0x00595f thru 0x01506b (0xf70d bytes) 000566 1763876617.041062 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9f5499f0 (stale) Mirror 10: 0x9f5499f0 (stale) Mirror 12: 0x9f5499f0 (stale) Mirror 13: 0x9f5499f0 (stale) Mirror 14: 0x9f5499f0 (stale) Mirror 15: 0x9f5499f0 (stale) Mirror 17: 0x9f5499f0 (stale) Mirror 18: 0x9f5499f0 (stale) Mirror 19: 0x9f5499f0 (stale) Mirror 21: 0x9f5499f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9f5499f0 (stale) Mirror 10: 0x9f5499f0 (stale) Mirror 12: 0x9f5499f0 (stale) Mirror 13: 0x9f5499f0 (stale) Mirror 14: 0x9f5499f0 (stale) Mirror 15: 0x9f5499f0 (stale) Mirror 17: 0x9f5499f0 (stale) Mirror 18: 0x9f5499f0 (stale) Mirror 19: 0x9f5499f0 (stale) Mirror 21: 0x9f5499f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000567 1763876617.186648 write 0x03015f thru 0x038f50 (0x8df2 bytes) 000568 1763876617.190859 punch from 0x01c606 to 0x026dc6 (0xa7c1 bytes) 000569 1763876617.192535 fallocate 0x038574 thru 0x03ffff (0x7a8c bytes) 000571 1763876617.194473 write 0x027398 thru 0x02cf18 (0x5b81 bytes) 000572 1763876617.197193 fallocate 0x03968c thru 0x03ffff (0x6974 bytes) 000573 1763876617.198923 write 0x01bc3f thru 0x023be8 (0x7faa bytes) 000574 1763876617.201684 fallocate 0x02a619 thru 0x037b54 (0xd53c bytes) 000575 1763876617.203035 mapread 0x020338 thru 0x02a09c (0x9d65 bytes) 000576 1763876617.203562 mapwrite 0x037849 thru 0x03ffff (0x87b7 bytes) 000577 1763876617.211615 trunc from 0x01c4c2 to 0x02b796 (0xf2d5 bytes) 000578 1763876617.213402 read 0x03352d thru 0x03ffff (0xcad3 bytes) 000579 1763876617.213734 trunc from 0x040000 to 0x030993 (0xf66c bytes) 000581 1763876617.218655 trunc from 0x0198d8 to 0x0276f2 (0xde1b bytes) 000582 1763876617.220426 trunc from 0x029d2b to 0x02caf3 (0x2dc9 bytes) 000584 1763876617.222355 write 0x00dad0 thru 0x0178c0 (0x9df1 bytes) 000585 1763876617.225864 fallocate 0x022746 thru 0x0299cf (0x728a bytes) 000586 1763876617.228818 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x30994): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30994): Mirror 6: 0xe088ac4f (stale) Mirror 10: 0xe088ac4f (stale) Mirror 12: 0xe088ac4f (stale) Mirror 13: 0xe088ac4f (stale) Mirror 14: 0xe088ac4f (stale) Mirror 15: 0xe088ac4f (stale) Mirror 17: 0xe088ac4f (stale) Mirror 18: 0xe088ac4f (stale) Mirror 19: 0xe088ac4f (stale) Mirror 21: 0xe088ac4f (stale) CRC-32 checksum value for chunk [0, 0x30994): Mirror 1: 0xc5ef3778 Mirror 22: 0xc5ef3778 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30994: skipped 000587 1763876617.481055 mapwrite 0x023fb1 thru 0x031648 (0xd698 bytes) 000588 1763876617.507824 fallocate 0x01254e thru 0x01ded6 (0xb989 bytes) 000589 1763876617.509609 trunc from 0x031649 to 0x0374f5 (0x5ead bytes) 000590 1763876617.514176 mapread 0x004ca2 thru 0x00fc9e (0xaffd bytes) 000592 1763876617.518115 mapread 0x001b77 thru 0x00c9e4 (0xae6e bytes) 000593 1763876617.518982 mapwrite 0x01dfeb thru 0x02be0f (0xde25 bytes) 000594 1763876617.526659 write 0x022d8c thru 0x0280ef (0x5364 bytes) 000595 1763876617.528636 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x374f6): Mirror 6: 0xaa234f25 (stale) Mirror 10: 0xaa234f25 (stale) Mirror 12: 0xaa234f25 (stale) Mirror 14: 0xaa234f25 (stale) Mirror 15: 0xaa234f25 (stale) Mirror 17: 0xaa234f25 (stale) Mirror 18: 0xaa234f25 (stale) Mirror 19: 0xaa234f25 (stale) Mirror 21: 0xaa234f25 (stale) Mirror 22: 0x8cb961b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x374f6): Mirror 6: 0xaa234f25 (stale) Mirror 10: 0xaa234f25 (stale) Mirror 12: 0xaa234f25 (stale) Mirror 14: 0xaa234f25 (stale) Mirror 15: 0xaa234f25 (stale) Mirror 17: 0xaa234f25 (stale) Mirror 18: 0xaa234f25 (stale) Mirror 19: 0xaa234f25 (stale) Mirror 21: 0xaa234f25 (stale) Mirror 22: 0x8cb961b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x374f6: skipped 000596 1763876617.671772 write 0x021888 thru 0x021c0d (0x386 bytes) 000597 1763876617.674430 read 0x00146c thru 0x00aac8 (0x965d bytes) 000598 1763876617.676910 mapwrite 0x005478 thru 0x012d70 (0xd8f9 bytes) 000600 1763876617.686359 mapwrite 0x000644 thru 0x000e87 (0x844 bytes) 000601 1763876617.690108 mapread 0x00a6ff thru 0x00cb7a (0x247c bytes) 000602 1763876617.691425 read 0x02113f thru 0x02ea9d (0xd95f bytes) 000603 1763876617.692029 read 0x030882 thru 0x036fa1 (0x6720 bytes) 000605 1763876617.693047 read 0x015b89 thru 0x022c26 (0xd09e bytes) 000606 1763876617.693667 trunc from 0x0374f6 to 0x03a37b (0x2e86 bytes) 000607 1763876617.698228 fallocate 0x00e39a thru 0x018298 (0x9eff bytes) 000608 1763876617.700150 write 0x01722d thru 0x0174d1 (0x2a5 bytes) 000609 1763876617.700753 write 0x006051 thru 0x009d98 (0x3d48 bytes) 000610 1763876617.701543 write 0x008fde thru 0x011db8 (0x8ddb bytes) 000611 1763876617.702185 write 0x01d9f2 thru 0x02a50d (0xcb1c bytes) 000612 1763876617.703349 trunc from 0x03a37c to 0x027ece (0x124ad bytes) 000613 1763876617.708127 write 0x02e388 thru 0x03b85f (0xd4d8 bytes) 000614 1763876617.709176 trunc from 0x03b860 to 0x019d80 (0x21adf bytes) 000615 1763876617.713894 fallocate 0x022a69 thru 0x031300 (0xe898 bytes) 000616 1763876617.715880 fallocate 0x02a52b thru 0x02b1df (0xcb5 bytes) 000617 1763876617.717656 read 0x018ba5 thru 0x019d80 (0x11dc bytes) 000618 1763876617.717964 read 0x0061d4 thru 0x015610 (0xf43d bytes) 000619 1763876617.718388 mapwrite 0x015975 thru 0x0226ec (0xcd78 bytes) 000620 1763876617.733751 MIRROR_RESYNC 11 mirrors 000621 1763876618.078791 write 0x039b5e thru 0x03ccb3 (0x3156 bytes) 000624 1763876618.089778 write 0x003e26 thru 0x00a54f (0x672a bytes) 000625 1763876618.092790 punch from 0x01fd7b to 0x023b62 (0x3de8 bytes) 000626 1763876618.094153 write 0x002199 thru 0x010f89 (0xedf1 bytes) 000627 1763876618.097070 trunc from 0x02659a to 0x02a6f5 (0x415c bytes) 000628 1763876618.098658 write 0x010f98 thru 0x01825e (0x72c7 bytes) 000629 1763876618.100632 write 0x0196c7 thru 0x026037 (0xc971 bytes) 000630 1763876618.105415 read 0x033006 thru 0x033269 (0x264 bytes) 000631 1763876618.107088 mapread 0x0352f1 thru 0x035a3b (0x74b bytes) 000632 1763876618.109997 mapwrite 0x02f1d0 thru 0x03a9d3 (0xb804 bytes) 000633 1763876618.120740 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3ccb4): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ccb4): Mirror 6: 0xf491652d (stale) Mirror 10: 0xf491652d (stale) Mirror 12: 0xf491652d (stale) Mirror 14: 0xf491652d (stale) Mirror 15: 0xf491652d (stale) Mirror 17: 0xf491652d (stale) Mirror 18: 0xf491652d (stale) Mirror 19: 0xf491652d (stale) Mirror 21: 0xf491652d (stale) Mirror 22: 0xf491652d (stale) CRC-32 checksum value for chunk [0, 0x3ccb4): Mirror 1: 0x57b1c4bc Mirror 23: 0x57b1c4bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ccb4: skipped 000634 1763876618.391747 mapwrite 0x0232be thru 0x02445c (0x119f bytes) 000635 1763876618.427008 read 0x027011 thru 0x032af1 (0xbae1 bytes) 000636 1763876618.427645 write 0x03ec21 thru 0x03ffff (0x13df bytes) 000637 1763876618.429691 fallocate 0x01f4eb thru 0x02db3a (0xe650 bytes) 000638 1763876618.431905 mapread 0x03b6f2 thru 0x03ffff (0x490e bytes) 000639 1763876618.434537 trunc from 0x013fc0 to 0x0162e0 (0x2321 bytes) 000641 1763876618.436944 trunc from 0x0123f7 to 0x0191a0 (0x6daa bytes) 000642 1763876618.438788 read 0x02931e thru 0x037da0 (0xea83 bytes) 000643 1763876618.439346 write 0x0186d1 thru 0x01d7ee (0x511e bytes) 000644 1763876618.442372 write 0x020b32 thru 0x0283bc (0x788b bytes) 000645 1763876618.443220 trunc from 0x02a40c to 0x036417 (0xc00c bytes) 000646 1763876618.445421 trunc from 0x00e666 to 0x0100ae (0x1a49 bytes) 000647 1763876618.447346 write 0x02d7d1 thru 0x032649 (0x4e79 bytes) 000648 1763876618.450080 mapwrite 0x01f0ec thru 0x02c39a (0xd2af bytes) 000649 1763876618.458565 write 0x0211ed thru 0x031111 (0xff25 bytes) 000650 1763876618.459435 trunc from 0x040000 to 0x016fbd (0x29042 bytes) 000651 1763876618.469733 write 0x0236e8 thru 0x0307f4 (0xd10d bytes) 000652 1763876618.471941 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x307f5): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x307f5): Mirror 6: 0x6db8f75f (stale) Mirror 10: 0x6db8f75f (stale) Mirror 12: 0x6db8f75f (stale) Mirror 14: 0x6db8f75f (stale) Mirror 15: 0x6db8f75f (stale) Mirror 17: 0x6db8f75f (stale) Mirror 18: 0x6db8f75f (stale) Mirror 19: 0x6db8f75f (stale) Mirror 21: 0x6db8f75f (stale) Mirror 22: 0x6db8f75f (stale) Mirror 23: 0x1a62775d (stale) CRC-32 checksum value for chunk [0, 0x307f5): Mirror 1: 0xae13cc6c Mirror 24: 0xae13cc6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x307f5: skipped 000654 1763876618.741702 punch from 0x01b07d to 0x02619e (0xb122 bytes) 000655 1763876618.754527 write 0x01598a thru 0x01a0ea (0x4761 bytes) 000657 1763876618.759230 read 0x007c3f thru 0x00f5f2 (0x79b4 bytes) 000659 1763876618.762211 mapread 0x009724 thru 0x00f17b (0x5a58 bytes) 000660 1763876618.762840 write 0x003112 thru 0x005271 (0x2160 bytes) 000661 1763876618.765819 trunc from 0x02c3ab to 0x0307f4 (0x444a bytes) 000662 1763876618.767807 read 0x027108 thru 0x02c2da (0x51d3 bytes) 000663 1763876618.771447 fallocate 0x000a83 thru 0x00c686 (0xbc04 bytes) 000664 1763876618.773968 read 0x00393f thru 0x00d563 (0x9c25 bytes) 000665 1763876618.774559 mapread 0x00f769 thru 0x010ee7 (0x177f bytes) 000666 1763876618.775401 mapread 0x02c62d thru 0x0307f4 (0x41c8 bytes) 000669 1763876618.776948 fallocate 0x027b99 thru 0x02eccc (0x7134 bytes) 000670 1763876618.802778 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x307f5): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x307f5): Mirror 6: 0x6db8f75f (stale) Mirror 10: 0x6db8f75f (stale) Mirror 12: 0x6db8f75f (stale) Mirror 14: 0x6db8f75f (stale) Mirror 15: 0x6db8f75f (stale) Mirror 17: 0x6db8f75f (stale) Mirror 18: 0x6db8f75f (stale) Mirror 19: 0x6db8f75f (stale) Mirror 21: 0x6db8f75f (stale) Mirror 22: 0x6db8f75f (stale) Mirror 23: 0x1a62775d (stale) Mirror 24: 0xae13cc6c (stale) CRC-32 checksum value for chunk [0, 0x307f5): Mirror 1: 0x22915cd7 Mirror 25: 0x22915cd7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x307f5: skipped 000671 1763876619.108613 trunc from 0x0307f5 to 0x00bb9a (0x24c5a bytes) 000672 1763876619.125992 write 0x038659 thru 0x03d56f (0x4f17 bytes) 000673 1763876619.126862 read 0x01ac1b thru 0x028d24 (0xe10a bytes) 000674 1763876619.129644 mapwrite 0x01eefb thru 0x021f7c (0x3082 bytes) 000675 1763876619.135651 mapread 0x00c07a thru 0x0147ef (0x8776 bytes) 000676 1763876619.139861 mapread 0x016d08 thru 0x01b122 (0x441b bytes) 000677 1763876619.140436 trunc from 0x015f75 to 0x02011c (0xa1a8 bytes) 000678 1763876619.142607 mapread 0x01d51a thru 0x025fc0 (0x8aa7 bytes) 000679 1763876619.145322 trunc from 0x03d570 to 0x012340 (0x2b22f bytes) 000680 1763876619.150749 mapwrite 0x0003b6 thru 0x00256e (0x21b9 bytes) 000681 1763876619.155571 fallocate 0x0201fe thru 0x029d71 (0x9b74 bytes) 000682 1763876619.157671 trunc from 0x021da2 to 0x0227cc (0xa2b bytes) 000683 1763876619.159204 mapwrite 0x037dec thru 0x03b573 (0x3788 bytes) 000684 1763876619.171135 trunc from 0x03b574 to 0x034a61 (0x6b12 bytes) 000685 1763876619.174879 trunc from 0x0241ad to 0x02b9c3 (0x7817 bytes) 000687 1763876619.176904 write 0x0008aa thru 0x00c87b (0xbfd2 bytes) 000688 1763876619.177692 write 0x020efe thru 0x02a36c (0x946f bytes) 000689 1763876619.179727 trunc from 0x034a62 to 0x0000de (0x34983 bytes) 000690 1763876619.184073 mapread 0x0000a4 thru 0x0000de (0x3b bytes) 000691 1763876619.184608 punch from 0x000031 to 0x0000de (0xae bytes) 000692 1763876619.188038 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdf): Mirror 6: 0xf6a4c443 (stale) Mirror 10: 0xf6a4c443 (stale) Mirror 12: 0xf6a4c443 (stale) Mirror 15: 0xf6a4c443 (stale) Mirror 17: 0xf6a4c443 (stale) Mirror 18: 0xf6a4c443 (stale) Mirror 19: 0xf6a4c443 (stale) Mirror 21: 0xf6a4c443 (stale) Mirror 22: 0xf6a4c443 (stale) Mirror 23: 0xf6a4c443 (stale) Mirror 24: 0xf6a4c443 (stale) Mirror 25: 0xf6a4c443 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdf): Mirror 6: 0xf6a4c443 (stale) Mirror 10: 0xf6a4c443 (stale) Mirror 12: 0xf6a4c443 (stale) Mirror 15: 0xf6a4c443 (stale) Mirror 17: 0xf6a4c443 (stale) Mirror 18: 0xf6a4c443 (stale) Mirror 19: 0xf6a4c443 (stale) Mirror 21: 0xf6a4c443 (stale) Mirror 22: 0xf6a4c443 (stale) Mirror 23: 0xf6a4c443 (stale) Mirror 24: 0xf6a4c443 (stale) Mirror 25: 0xf6a4c443 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdf: skipped 000693 1763876619.287403 trunc from 0x0000df to 0x01c770 (0x1c692 bytes) 000694 1763876619.293104 MIRROR_RESYNC 13 mirrors 000695 1763876619.643759 trunc from 0x01c771 to 0x010ef0 (0xb880 bytes) 000696 1763876619.659896 trunc from 0x010ef1 to 0x01500b (0x411b bytes) 000697 1763876619.664787 mapwrite 0x00d1a7 thru 0x016e8f (0x9ce9 bytes) 000700 1763876619.676131 mapwrite 0x0293ae thru 0x02bb2f (0x2782 bytes) 000701 1763876619.686401 mapread 0x015754 thru 0x01f50a (0x9db7 bytes) 000702 1763876619.688370 MIRROR_RESYNC 13 mirrors 000703 1763876620.165352 trunc from 0x02bb30 to 0x00bba9 (0x1ff86 bytes) 000705 1763876620.177537 trunc from 0x00bbaa to 0x03b8ff (0x2fd56 bytes) 000707 1763876620.182419 punch from 0x02bb89 to 0x03029b (0x4713 bytes) 000709 1763876620.184372 fallocate 0x017bd3 thru 0x02180a (0x9c38 bytes) 000710 1763876620.185934 mapread 0x0136f2 thru 0x01b233 (0x7b42 bytes) 000711 1763876620.190009 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3b900): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b900): Mirror 6: 0x54dd4b38 (stale) Mirror 10: 0x54dd4b38 (stale) Mirror 12: 0x54dd4b38 (stale) Mirror 15: 0x54dd4b38 (stale) Mirror 17: 0x54dd4b38 (stale) Mirror 18: 0x54dd4b38 (stale) Mirror 19: 0x54dd4b38 (stale) Mirror 21: 0x54dd4b38 (stale) Mirror 22: 0x54dd4b38 (stale) Mirror 23: 0x54dd4b38 (stale) Mirror 24: 0x54dd4b38 (stale) Mirror 25: 0x54dd4b38 (stale) CRC-32 checksum value for chunk [0, 0x3b900): Mirror 1: 0xa1764a83 Mirror 26: 0xa1764a83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b900: skipped 000713 1763876620.435661 mapwrite 0x02c7d0 thru 0x036ede (0xa70f bytes) 000714 1763876620.476428 trunc from 0x016c68 to 0x018c68 (0x2001 bytes) 000715 1763876620.480290 MIRROR_RESYNC 14 mirrors 000716 1763876620.978865 trunc from 0x00a48a to 0x00d253 (0x2dca bytes) 000717 1763876620.990529 fallocate 0x02feaf thru 0x03549c (0x55ee bytes) 000718 1763876620.992387 read 0x01bc5a thru 0x0266dd (0xaa84 bytes) 000719 1763876620.996624 MIRROR_RESYNC 14 mirrors 000720 1763876621.512654 read 0x0354b7 thru 0x03b8ff (0x6449 bytes) 000723 1763876621.515639 write 0x03dc0f thru 0x03ffff (0x23f1 bytes) 000724 1763876621.527655 punch from 0x021619 to 0x02d418 (0xbe00 bytes) 000725 1763876621.529822 mapwrite 0x01d77b thru 0x02ad06 (0xd58c bytes) 000726 1763876621.542298 mapwrite 0x025647 thru 0x031e63 (0xc81d bytes) 000727 1763876621.550282 punch from 0x010fff to 0x01aea6 (0x9ea8 bytes) 000728 1763876621.552213 write 0x002c88 thru 0x00f60a (0xc983 bytes) 000729 1763876621.553944 write 0x0087d7 thru 0x00c3ef (0x3c19 bytes) 000731 1763876621.555211 write 0x00d3b1 thru 0x00fa40 (0x2690 bytes) 000732 1763876621.555826 punch from 0x00ba19 to 0x01318b (0x7773 bytes) 000733 1763876621.559991 fallocate 0x02de5a thru 0x0301f1 (0x2398 bytes) 000734 1763876621.561804 trunc from 0x040000 to 0x036d57 (0x92a8 bytes) 000735 1763876621.567379 mapread 0x00bc04 thru 0x00c8ec (0xce9 bytes) 000736 1763876621.570744 trunc from 0x005626 to 0x00f687 (0xa062 bytes) 000737 1763876621.572668 trunc from 0x0267f6 to 0x027464 (0xc6f bytes) 000738 1763876621.575604 MIRROR_RESYNC 14 mirrors 000739 1763876622.083792 read 0x02238b thru 0x0285f9 (0x626f bytes) 000740 1763876622.085887 read 0x007c72 thru 0x01606b (0xe3fa bytes) 000741 1763876622.090239 MIRROR_EXTEND to 15 mirrors 000742 1763876622.294327 trunc from 0x036d58 to 0x021eff (0x14e58 bytes) 000743 1763876622.314943 fallocate 0x001331 thru 0x006740 (0x5410 bytes) 000744 1763876622.317048 punch from 0x01f1ad to 0x01f558 (0x3ac bytes) 000745 1763876622.318737 punch from 0x00bfb8 to 0x010ada (0x4b23 bytes) 000746 1763876622.320326 trunc from 0x021f00 to 0x03914d (0x1724e bytes) 000747 1763876622.327075 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3914e): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3914e): Mirror 6: 0x5b00877a (stale) Mirror 10: 0x5b00877a (stale) Mirror 12: 0x5b00877a (stale) Mirror 15: 0x5b00877a (stale) Mirror 17: 0x5b00877a (stale) Mirror 18: 0x5b00877a (stale) Mirror 19: 0x5b00877a (stale) Mirror 21: 0x5b00877a (stale) Mirror 22: 0x5b00877a (stale) Mirror 23: 0x5b00877a (stale) Mirror 24: 0x5b00877a (stale) Mirror 25: 0x5b00877a (stale) Mirror 26: 0x5b00877a (stale) Mirror 27: 0x5b00877a (stale) CRC-32 checksum value for chunk [0, 0x3914e): Mirror 1: 0xe7f65ed0 Mirror 28: 0xe7f65ed0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3914e: skipped 000748 1763876622.632133 trunc from 0x03914e to 0x0063e1 (0x32d6c bytes) 000749 1763876622.649742 trunc from 0x0063e2 to 0x02fdc7 (0x299e6 bytes) 000750 1763876622.655800 mapwrite 0x025fec thru 0x029755 (0x376a bytes) 000751 1763876622.663767 mapread 0x014198 thru 0x017f15 (0x3d7e bytes) 000752 1763876622.666989 fallocate 0x01f999 thru 0x02060f (0xc77 bytes) 000753 1763876622.668847 mapread 0x005899 thru 0x00c867 (0x6fcf bytes) 000754 1763876622.670878 MIRROR_RESYNC 16 mirrors 000755 1763876623.459977 write 0x00f2e3 thru 0x01654d (0x726b bytes) 000756 1763876623.473041 punch from 0x0152d5 to 0x02170e (0xc43a bytes) 000757 1763876623.476906 mapwrite 0x0036a8 thru 0x00cb6e (0x94c7 bytes) 000758 1763876623.486853 mapread 0x02e60b thru 0x02fdc7 (0x17bd bytes) 000759 1763876623.487741 mapwrite 0x020121 thru 0x023ca7 (0x3b87 bytes) 000760 1763876623.493601 fallocate 0x022aa7 thru 0x02ae20 (0x837a bytes) 000761 1763876623.496481 MIRROR_RESYNC 16 mirrors 000762 1763876624.500541 punch from 0x017f40 to 0x025c17 (0xdcd8 bytes) 000763 1763876624.512598 MIRROR_SPLIT mirror 8 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fdc8): Mirror 6: 0x25a0329 (stale) Mirror 10: 0x25a0329 (stale) Mirror 12: 0x25a0329 (stale) Mirror 15: 0x25a0329 (stale) Mirror 17: 0x25a0329 (stale) Mirror 18: 0x25a0329 (stale) Mirror 19: 0x25a0329 (stale) Mirror 22: 0x25a0329 (stale) Mirror 23: 0x25a0329 (stale) Mirror 24: 0x25a0329 (stale) Mirror 25: 0x25a0329 (stale) Mirror 26: 0x25a0329 (stale) Mirror 27: 0x25a0329 (stale) Mirror 28: 0x25a0329 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fdc8): Mirror 6: 0x25a0329 (stale) Mirror 10: 0x25a0329 (stale) Mirror 12: 0x25a0329 (stale) Mirror 15: 0x25a0329 (stale) Mirror 17: 0x25a0329 (stale) Mirror 18: 0x25a0329 (stale) Mirror 19: 0x25a0329 (stale) Mirror 22: 0x25a0329 (stale) Mirror 23: 0x25a0329 (stale) Mirror 24: 0x25a0329 (stale) Mirror 25: 0x25a0329 (stale) Mirror 26: 0x25a0329 (stale) Mirror 27: 0x25a0329 (stale) Mirror 28: 0x25a0329 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fdc8: skipped 000764 1763876624.709654 fallocate 0x00d69f thru 0x00eda8 (0x170a bytes) 000765 1763876624.711584 punch from 0x01ead0 to 0x02d801 (0xed32 bytes) 000766 1763876624.712932 mapwrite 0x01c9aa thru 0x01f56b (0x2bc2 bytes) 000767 1763876624.720322 trunc from 0x02fdc8 to 0x009aef (0x262d8 bytes) 000768 1763876624.725809 trunc from 0x009af0 to 0x0221e3 (0x186f4 bytes) 000769 1763876624.729822 read 0x0028a7 thru 0x004ab7 (0x2211 bytes) 000770 1763876624.733137 write 0x0118dc thru 0x020a42 (0xf167 bytes) 000771 1763876624.734538 trunc from 0x0221e4 to 0x02776b (0x5588 bytes) 000773 1763876624.739391 trunc from 0x0215ac to 0x02776b (0x61c0 bytes) 000774 1763876624.741083 read 0x023e8d thru 0x026c2c (0x2da0 bytes) 000776 1763876624.744221 mapwrite 0x02d69c thru 0x03d06b (0xf9d0 bytes) 000777 1763876624.759050 punch from 0x0281ab to 0x03202b (0x9e81 bytes) 000778 1763876624.760954 write 0x0365fd thru 0x03ffff (0x9a03 bytes) 000780 1763876624.761931 mapwrite 0x00b2e1 thru 0x012f21 (0x7c41 bytes) 000781 1763876624.768739 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6cb1bb61 (stale) Mirror 10: 0x6cb1bb61 (stale) Mirror 12: 0x6cb1bb61 (stale) Mirror 15: 0x6cb1bb61 (stale) Mirror 17: 0x6cb1bb61 (stale) Mirror 18: 0x6cb1bb61 (stale) Mirror 19: 0x6cb1bb61 (stale) Mirror 22: 0x6cb1bb61 (stale) Mirror 23: 0x6cb1bb61 (stale) Mirror 24: 0x6cb1bb61 (stale) Mirror 25: 0x6cb1bb61 (stale) Mirror 26: 0x6cb1bb61 (stale) Mirror 27: 0x6cb1bb61 (stale) Mirror 28: 0x6cb1bb61 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x38cbe1ba Mirror 29: 0x38cbe1ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000782 1763876625.072940 trunc from 0x040000 to 0x021aca (0x1e535 bytes) 000783 1763876625.093694 write 0x035959 thru 0x03ffff (0xa6a7 bytes) 000786 1763876625.095516 trunc from 0x040000 to 0x03bfcb (0x4034 bytes) 000787 1763876625.101489 mapread 0x02c528 thru 0x032c21 (0x66fa bytes) 000788 1763876625.104560 write 0x0011be thru 0x00ac44 (0x9a87 bytes) 000789 1763876625.107737 trunc from 0x03bfcc to 0x02ae0b (0x111c0 bytes) 000790 1763876625.113164 write 0x0141bb thru 0x01bdd7 (0x7c1d bytes) 000791 1763876625.116930 mapread 0x007a1a thru 0x00bfc5 (0x45ac bytes) 000792 1763876625.117636 read 0x00a6d8 thru 0x012588 (0x7eb1 bytes) 000794 1763876625.118317 punch from 0x018478 to 0x02586a (0xd3f3 bytes) 000795 1763876625.124212 mapread 0x01c48d thru 0x023ee3 (0x7a57 bytes) 000796 1763876625.127608 trunc from 0x02ae0c to 0x01c4c5 (0xe946 bytes) 000797 1763876625.133115 MIRROR_RESYNC 16 mirrors 000798 1763876625.587399 mapwrite 0x00a010 thru 0x013565 (0x9556 bytes) 000799 1763876625.622480 mapread 0x0072a5 thru 0x015d3f (0xea9b bytes) 000800 1763876625.623433 punch from 0x0028a2 to 0x00f6ab (0xce0a bytes) 000802 1763876625.626507 trunc from 0x01c4c6 to 0x0052d5 (0x171f0 bytes) 000803 1763876625.632436 mapread 0x002ab6 thru 0x0052d5 (0x2820 bytes) 000804 1763876625.634457 mapread 0x004b04 thru 0x0052d5 (0x7d2 bytes) 000805 1763876625.635121 write 0x036379 thru 0x038bf9 (0x2881 bytes) 000806 1763876625.635844 trunc from 0x038bfa to 0x00eeea (0x29d0f bytes) 000807 1763876625.640294 trunc from 0x00eeeb to 0x016949 (0x7a5f bytes) 000808 1763876625.644468 write 0x01785c thru 0x01997f (0x2124 bytes) 000809 1763876625.645215 write 0x021f23 thru 0x028750 (0x682e bytes) 000810 1763876625.646050 write 0x005a79 thru 0x009bbd (0x4145 bytes) 000811 1763876625.648909 trunc from 0x028751 to 0x02379b (0x4fb5 bytes) 000813 1763876625.656459 mapwrite 0x005c64 thru 0x0131cf (0xd56c bytes) 000814 1763876625.667996 punch from 0x00afcc to 0x012fe7 (0x801c bytes) 000815 1763876625.670285 fallocate 0x00247f thru 0x00d56a (0xb0ec bytes) 000816 1763876625.672411 write 0x01dc01 thru 0x0288b3 (0xacb3 bytes) 000817 1763876625.673564 punch from 0x00d77a to 0x014fd2 (0x7859 bytes) 000818 1763876625.675421 mapread 0x027c42 thru 0x0288b3 (0xc72 bytes) 000819 1763876625.676178 write 0x0202a3 thru 0x02a359 (0xa0b7 bytes) 000820 1763876625.677355 read 0x02097d thru 0x02a359 (0x99dd bytes) 000821 1763876625.677847 punch from 0x020b4a to 0x021316 (0x7cd bytes) 000823 1763876625.682894 fallocate 0x0204c8 thru 0x029c31 (0x976a bytes) 000824 1763876625.684862 mapread 0x0254bb thru 0x02a359 (0x4e9f bytes) 000825 1763876625.685693 fallocate 0x01e1b3 thru 0x02b87d (0xd6cb bytes) 000826 1763876625.687699 write 0x005c91 thru 0x00b8e4 (0x5c54 bytes) 000827 1763876625.689958 fallocate 0x006bab thru 0x00d640 (0x6a96 bytes) 000828 1763876625.691672 write 0x012e00 thru 0x013acf (0xcd0 bytes) 000830 1763876625.694564 fallocate 0x004748 thru 0x005c8f (0x1548 bytes) 000833 1763876625.696930 mapwrite 0x003d2d thru 0x011014 (0xd2e8 bytes) 000834 1763876625.704236 fallocate 0x00c59e thru 0x01482e (0x8291 bytes) 000835 1763876625.705872 trunc from 0x0155ce to 0x022176 (0xcba9 bytes) 000837 1763876625.707796 fallocate 0x01563f thru 0x02228f (0xcc51 bytes) 000838 1763876625.709395 trunc from 0x006de2 to 0x016059 (0xf278 bytes) 000839 1763876625.711003 mapread 0x02b664 thru 0x02b87d (0x21a bytes) 000840 1763876625.711727 fallocate 0x01c412 thru 0x029d47 (0xd936 bytes) 000843 1763876625.714431 write 0x01cc81 thru 0x0248fa (0x7c7a bytes) 000844 1763876625.716467 trunc from 0x02a42b to 0x02b87d (0x1453 bytes) 000845 1763876625.718210 trunc from 0x022e56 to 0x02ac74 (0x7e1f bytes) 000846 1763876625.721871 fallocate 0x01419c thru 0x01c4fe (0x8363 bytes) 000847 1763876625.723475 write 0x03db1a thru 0x03ffff (0x24e6 bytes) 000848 1763876625.724336 read 0x0155e6 thru 0x021ec5 (0xc8e0 bytes) 000849 1763876625.727743 fallocate 0x02c9c8 thru 0x031692 (0x4ccb bytes) 000850 1763876625.729266 read 0x01c99d thru 0x02c06d (0xf6d1 bytes) 000852 1763876625.730340 mapwrite 0x036d20 thru 0x03ffff (0x92e0 bytes) 000853 1763876625.736893 fallocate 0x031e5c thru 0x03ffff (0xe1a4 bytes) 000855 1763876625.741117 MIRROR_RESYNC 16 mirrors 000856 1763876626.473049 trunc from 0x040000 to 0x00953d (0x36ac2 bytes) 000858 1763876626.485232 mapread 0x006eea thru 0x0088a5 (0x19bc bytes) 000859 1763876626.487450 fallocate 0x0117ff thru 0x01a994 (0x9196 bytes) 000860 1763876626.489359 punch from 0x00d931 to 0x01a14b (0xc81b bytes) 000861 1763876626.490878 write 0x01a4a4 thru 0x024e41 (0xa99e bytes) 000862 1763876626.494141 punch from 0x02262b to 0x024e41 (0x2817 bytes) 000863 1763876626.498097 punch from 0x0222c6 to 0x0238f1 (0x162c bytes) 000864 1763876626.499641 mapread 0x001420 thru 0x005018 (0x3bf9 bytes) 000865 1763876626.502851 write 0x03a07f thru 0x03ffff (0x5f81 bytes) 000866 1763876626.503937 trunc from 0x02eb63 to 0x02eec3 (0x361 bytes) 000867 1763876626.505762 write 0x02bf0e thru 0x02ff03 (0x3ff6 bytes) 000868 1763876626.509119 write 0x028da7 thru 0x02fd5a (0x6fb4 bytes) 000869 1763876626.509911 trunc from 0x00d64f to 0x01bc12 (0xe5c4 bytes) 000870 1763876626.511805 mapwrite 0x021fd7 thru 0x02724d (0x5277 bytes) 000871 1763876626.518479 mapwrite 0x015098 thru 0x01751b (0x2484 bytes) 000872 1763876626.525744 trunc from 0x03bcff to 0x03ffff (0x4301 bytes) 000873 1763876626.527555 read 0x02732a thru 0x02f719 (0x83f0 bytes) 000874 1763876626.527968 mapread 0x00a933 thru 0x011be5 (0x72b3 bytes) 000875 1763876626.528627 mapread 0x00df74 thru 0x01371f (0x57ac bytes) 000876 1763876626.529268 write 0x010460 thru 0x011368 (0xf09 bytes) 000877 1763876626.530020 read 0x011b0f thru 0x01787e (0x5d70 bytes) 000878 1763876626.530538 mapwrite 0x013537 thru 0x020d65 (0xd82f bytes) 000879 1763876626.538821 mapwrite 0x02e4f0 thru 0x032766 (0x4277 bytes) 000880 1763876626.544996 read 0x00477f thru 0x0073df (0x2c61 bytes) 000881 1763876626.545400 trunc from 0x040000 to 0x036f31 (0x90ce bytes) 000882 1763876626.551532 write 0x008aaa thru 0x008cfb (0x252 bytes) 000883 1763876626.552062 mapread 0x036c2e thru 0x036f31 (0x304 bytes) 000884 1763876626.552821 trunc from 0x036f32 to 0x0050c2 (0x31e6f bytes) 000885 1763876626.557491 trunc from 0x0050c3 to 0x03951c (0x3445a bytes) 000886 1763876626.562055 write 0x027b7c thru 0x033eed (0xc372 bytes) 000887 1763876626.566365 fallocate 0x020894 thru 0x0226d3 (0x1e40 bytes) 000888 1763876626.568324 fallocate 0x010724 thru 0x01d40e (0xcceb bytes) 000889 1763876626.569747 trunc from 0x02ba3d to 0x037f1a (0xc4de bytes) 000890 1763876626.573882 write 0x000cb2 thru 0x008e2c (0x817b bytes) 000891 1763876626.574976 trunc from 0x03951d to 0x023b60 (0x159bc bytes) 000892 1763876626.579941 mapwrite 0x003a87 thru 0x009486 (0x5a00 bytes) 000893 1763876626.586181 punch from 0x0135cb to 0x016066 (0x2a9c bytes) 000894 1763876626.587773 trunc from 0x023b61 to 0x010b20 (0x13040 bytes) 000895 1763876626.591911 fallocate 0x01d5dd thru 0x0278d7 (0xa2fb bytes) 000896 1763876626.593535 mapread 0x00e4d9 thru 0x0170f0 (0x8c18 bytes) 000897 1763876626.596472 mapread 0x01dba8 thru 0x0278d7 (0x9d30 bytes) 000898 1763876626.597144 fallocate 0x0371e0 thru 0x03f6b9 (0x84da bytes) 000899 1763876626.599000 trunc from 0x01c20b to 0x02407b (0x7e71 bytes) 000901 1763876626.600714 read 0x019120 thru 0x025e55 (0xcd36 bytes) 000902 1763876626.603064 fallocate 0x034997 thru 0x03b1b2 (0x681c bytes) 000903 1763876626.604404 write 0x0254c1 thru 0x02e2d9 (0x8e19 bytes) 000904 1763876626.605311 write 0x01586b thru 0x018dfd (0x3593 bytes) 000905 1763876626.605891 fallocate 0x035eeb thru 0x03ffff (0xa115 bytes) 000906 1763876626.607657 punch from 0x00a19a to 0x018a21 (0xe888 bytes) 000907 1763876626.610962 mapread 0x01a6b0 thru 0x02625c (0xbbad bytes) 000908 1763876626.611528 mapwrite 0x037e24 thru 0x03ffff (0x81dc bytes) 000909 1763876626.621999 fallocate 0x02c296 thru 0x03c0a3 (0xfe0e bytes) 000910 1763876626.624598 MIRROR_RESYNC 16 mirrors 000912 1763876627.382141 trunc from 0x024e29 to 0x026fa6 (0x217e bytes) 000913 1763876627.393087 read 0x037575 thru 0x039d7a (0x2806 bytes) 000914 1763876627.395496 punch from 0x02f7bc to 0x031070 (0x18b5 bytes) 000915 1763876627.397280 mapwrite 0x03039c thru 0x03a164 (0x9dc9 bytes) 000916 1763876627.409275 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x24bb81cb (stale) Mirror 10: 0x24bb81cb (stale) Mirror 15: 0x24bb81cb (stale) Mirror 17: 0x24bb81cb (stale) Mirror 18: 0x24bb81cb (stale) Mirror 19: 0x24bb81cb (stale) Mirror 22: 0x24bb81cb (stale) Mirror 23: 0x24bb81cb (stale) Mirror 24: 0x24bb81cb (stale) Mirror 25: 0x24bb81cb (stale) Mirror 26: 0x24bb81cb (stale) Mirror 27: 0x24bb81cb (stale) Mirror 28: 0x24bb81cb (stale) Mirror 29: 0x24bb81cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x24bb81cb (stale) Mirror 10: 0x24bb81cb (stale) Mirror 15: 0x24bb81cb (stale) Mirror 17: 0x24bb81cb (stale) Mirror 18: 0x24bb81cb (stale) Mirror 19: 0x24bb81cb (stale) Mirror 22: 0x24bb81cb (stale) Mirror 23: 0x24bb81cb (stale) Mirror 24: 0x24bb81cb (stale) Mirror 25: 0x24bb81cb (stale) Mirror 26: 0x24bb81cb (stale) Mirror 27: 0x24bb81cb (stale) Mirror 28: 0x24bb81cb (stale) Mirror 29: 0x24bb81cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000917 1763876627.595821 write 0x0026d6 thru 0x008bd1 (0x64fc bytes) 000918 1763876627.599751 trunc from 0x040000 to 0x01d2f9 (0x22d06 bytes) 000919 1763876627.603820 punch from 0x012510 to 0x01425d (0x1d4e bytes) 000920 1763876627.605440 punch from 0x006ded to 0x0116a9 (0xa8bd bytes) 000921 1763876627.609041 read 0x00ec15 thru 0x00fe42 (0x122e bytes) 000922 1763876627.610982 mapread 0x0031fc thru 0x008294 (0x5099 bytes) 000923 1763876627.614158 read 0x01039b thru 0x01b4e1 (0xb147 bytes) 000925 1763876627.616511 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d2fa): Mirror 6: 0xbae92dce (stale) Mirror 10: 0xbae92dce (stale) Mirror 15: 0xbae92dce (stale) Mirror 17: 0xbae92dce (stale) Mirror 18: 0xbae92dce (stale) Mirror 19: 0xbae92dce (stale) Mirror 22: 0xbae92dce (stale) Mirror 23: 0xbae92dce (stale) Mirror 24: 0xbae92dce (stale) Mirror 25: 0xbae92dce (stale) Mirror 27: 0xbae92dce (stale) Mirror 28: 0xbae92dce (stale) Mirror 29: 0xbae92dce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d2fa): Mirror 6: 0xbae92dce (stale) Mirror 10: 0xbae92dce (stale) Mirror 15: 0xbae92dce (stale) Mirror 17: 0xbae92dce (stale) Mirror 18: 0xbae92dce (stale) Mirror 19: 0xbae92dce (stale) Mirror 22: 0xbae92dce (stale) Mirror 23: 0xbae92dce (stale) Mirror 24: 0xbae92dce (stale) Mirror 25: 0xbae92dce (stale) Mirror 27: 0xbae92dce (stale) Mirror 28: 0xbae92dce (stale) Mirror 29: 0xbae92dce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d2fa: skipped 000926 1763876627.816610 trunc from 0x01d2fa to 0x0283b7 (0xb0be bytes) 000927 1763876627.823782 fallocate 0x01571f thru 0x0160ed (0x9cf bytes) 000928 1763876627.826296 write 0x01d794 thru 0x02b912 (0xe17f bytes) 000929 1763876627.830492 trunc from 0x0050ca to 0x005bfe (0xb35 bytes) 000930 1763876627.833028 mapwrite 0x00aaee thru 0x01656e (0xba81 bytes) 000931 1763876627.847763 fallocate 0x038e6b thru 0x03ffff (0x7195 bytes) 000932 1763876627.850464 read 0x038330 thru 0x03ffff (0x7cd0 bytes) 000933 1763876627.854385 read 0x01634b thru 0x026205 (0xfebb bytes) 000934 1763876627.855010 read 0x00b50e thru 0x00d47b (0x1f6e bytes) 000935 1763876627.855521 read 0x03bbc9 thru 0x03e88c (0x2cc4 bytes) 000936 1763876627.855903 trunc from 0x01ea14 to 0x01f2c2 (0x8af bytes) 000937 1763876627.858221 read 0x014e6a thru 0x01d666 (0x87fd bytes) 000938 1763876627.858653 fallocate 0x032718 thru 0x036c7f (0x4568 bytes) 000939 1763876627.860647 trunc from 0x019a7d to 0x01d222 (0x37a6 bytes) 000940 1763876627.864024 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x24bb81cb (stale) Mirror 10: 0x24bb81cb (stale) Mirror 15: 0x24bb81cb (stale) Mirror 17: 0x24bb81cb (stale) Mirror 18: 0x24bb81cb (stale) Mirror 19: 0x24bb81cb (stale) Mirror 22: 0x24bb81cb (stale) Mirror 23: 0x24bb81cb (stale) Mirror 24: 0x24bb81cb (stale) Mirror 25: 0x24bb81cb (stale) Mirror 27: 0x24bb81cb (stale) Mirror 29: 0x24bb81cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x24bb81cb (stale) Mirror 10: 0x24bb81cb (stale) Mirror 15: 0x24bb81cb (stale) Mirror 17: 0x24bb81cb (stale) Mirror 18: 0x24bb81cb (stale) Mirror 19: 0x24bb81cb (stale) Mirror 22: 0x24bb81cb (stale) Mirror 23: 0x24bb81cb (stale) Mirror 24: 0x24bb81cb (stale) Mirror 25: 0x24bb81cb (stale) Mirror 27: 0x24bb81cb (stale) Mirror 29: 0x24bb81cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000941 1763876628.054007 mapread 0x03eb34 thru 0x03ffff (0x14cc bytes) 000942 1763876628.055889 fallocate 0x002816 thru 0x0104b1 (0xdc9c bytes) 000943 1763876628.057565 read 0x03298d thru 0x0396c4 (0x6d38 bytes) 000945 1763876628.060751 mapread 0x01d2af thru 0x02ca2f (0xf781 bytes) 000946 1763876628.061373 mapread 0x02f50b thru 0x03a879 (0xb36f bytes) 000947 1763876628.061886 trunc from 0x021296 to 0x022403 (0x116e bytes) 000948 1763876628.063433 write 0x02b1a0 thru 0x02f30f (0x4170 bytes) 000949 1763876628.066343 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x24bb81cb (stale) Mirror 10: 0x24bb81cb (stale) Mirror 15: 0x24bb81cb (stale) Mirror 17: 0x24bb81cb (stale) Mirror 18: 0x24bb81cb (stale) Mirror 19: 0x24bb81cb (stale) Mirror 22: 0x24bb81cb (stale) Mirror 23: 0x24bb81cb (stale) Mirror 24: 0x24bb81cb (stale) Mirror 25: 0x24bb81cb (stale) Mirror 27: 0x24bb81cb (stale) Mirror 29: 0x24bb81cb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x53b2113a Mirror 30: 0x53b2113a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000950 1763876628.404929 punch from 0x01f0fa to 0x02389b (0x47a2 bytes) 000951 1763876628.417453 mapread 0x00e160 thru 0x01cdcd (0xec6e bytes) 000952 1763876628.420916 write 0x03c767 thru 0x03ffff (0x3899 bytes) 000954 1763876628.423073 mapread 0x002167 thru 0x00be78 (0x9d12 bytes) 000955 1763876628.426629 read 0x02d072 thru 0x03a1d2 (0xd161 bytes) 000957 1763876628.427357 mapwrite 0x03e8e2 thru 0x03ffff (0x171e bytes) 000958 1763876628.431748 write 0x007f7a thru 0x014679 (0xc700 bytes) 000959 1763876628.433055 trunc from 0x040000 to 0x02425c (0x1bda3 bytes) 000960 1763876628.444506 mapread 0x01dfda thru 0x023858 (0x587f bytes) 000961 1763876628.445377 fallocate 0x00e667 thru 0x012c8c (0x4626 bytes) 000962 1763876628.449129 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x54:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2425d): Mirror 6: 0x8ff34d8f (stale) Mirror 10: 0x8ff34d8f (stale) Mirror 15: 0x8ff34d8f (stale) Mirror 17: 0x8ff34d8f (stale) Mirror 18: 0x8ff34d8f (stale) Mirror 19: 0x8ff34d8f (stale) Mirror 22: 0x8ff34d8f (stale) Mirror 23: 0x8ff34d8f (stale) Mirror 24: 0x8ff34d8f (stale) Mirror 27: 0x8ff34d8f (stale) Mirror 29: 0x8ff34d8f (stale) Mirror 30: 0x458894cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2425d): Mirror 6: 0x8ff34d8f (stale) Mirror 10: 0x8ff34d8f (stale) Mirror 15: 0x8ff34d8f (stale) Mirror 17: 0x8ff34d8f (stale) Mirror 18: 0x8ff34d8f (stale) Mirror 19: 0x8ff34d8f (stale) Mirror 22: 0x8ff34d8f (stale) Mirror 23: 0x8ff34d8f (stale) Mirror 24: 0x8ff34d8f (stale) Mirror 27: 0x8ff34d8f (stale) Mirror 29: 0x8ff34d8f (stale) Mirror 30: 0x458894cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2425d: skipped 000964 1763876628.638794 trunc from 0x02425d to 0x018cfa (0xb562 bytes) 000965 1763876628.643584 trunc from 0x018cfb to 0x027987 (0xec8d bytes) 000966 1763876628.646949 mapwrite 0x01c608 thru 0x026f0d (0xa906 bytes) 000967 1763876628.656990 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x27988): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27988): Mirror 6: 0x18ddce98 (stale) Mirror 10: 0x18ddce98 (stale) Mirror 15: 0x18ddce98 (stale) Mirror 17: 0x18ddce98 (stale) Mirror 18: 0x18ddce98 (stale) Mirror 19: 0x18ddce98 (stale) Mirror 22: 0x18ddce98 (stale) Mirror 23: 0x18ddce98 (stale) Mirror 24: 0x18ddce98 (stale) Mirror 27: 0x18ddce98 (stale) Mirror 29: 0x18ddce98 (stale) Mirror 30: 0x1eaf7de4 (stale) CRC-32 checksum value for chunk [0, 0x27988): Mirror 1: 0x72af16f2 Mirror 31: 0x72af16f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27988: skipped 000968 1763876628.932029 trunc from 0x019daf to 0x01d3a2 (0x35f4 bytes) 000969 1763876628.945151 write 0x006d55 thru 0x009e5d (0x3109 bytes) 000970 1763876628.949227 write 0x02f20c thru 0x0391f9 (0x9fee bytes) 000971 1763876628.950389 read 0x02b9bc thru 0x034d2c (0x9371 bytes) 000972 1763876628.952385 mapwrite 0x02cc7b thru 0x039a06 (0xcd8c bytes) 000973 1763876628.968767 trunc from 0x039a07 to 0x0057df (0x34227 bytes) 000974 1763876628.973531 mapwrite 0x029064 thru 0x039016 (0xffb3 bytes) 000975 1763876628.987050 fallocate 0x0298d1 thru 0x02e559 (0x4c89 bytes) 000976 1763876628.988741 mapread 0x0341ff thru 0x039016 (0x4e18 bytes) 000977 1763876628.989475 read 0x03573e thru 0x036271 (0xb34 bytes) 000978 1763876628.989770 punch from 0x00c3ba to 0x0125e3 (0x622a bytes) 000980 1763876628.992065 trunc from 0x02844d to 0x02aaa1 (0x2655 bytes) 000981 1763876628.993785 write 0x01c443 thru 0x022ae5 (0x66a3 bytes) 000982 1763876628.994595 trunc from 0x039017 to 0x035f84 (0x3092 bytes) 000983 1763876628.999231 mapread 0x003081 thru 0x00dded (0xad6d bytes) 000984 1763876629.002379 read 0x00276e thru 0x006ca4 (0x4537 bytes) 000985 1763876629.002713 write 0x0006d5 thru 0x00d39b (0xccc7 bytes) 000986 1763876629.004949 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x35f85): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f85): Mirror 6: 0xbf7a0286 (stale) Mirror 10: 0xbf7a0286 (stale) Mirror 15: 0xbf7a0286 (stale) Mirror 17: 0xbf7a0286 (stale) Mirror 18: 0xbf7a0286 (stale) Mirror 19: 0xbf7a0286 (stale) Mirror 22: 0xbf7a0286 (stale) Mirror 23: 0xbf7a0286 (stale) Mirror 24: 0xbf7a0286 (stale) Mirror 27: 0xbf7a0286 (stale) Mirror 29: 0xbf7a0286 (stale) Mirror 30: 0x283a796b (stale) Mirror 31: 0xbf2e587d (stale) CRC-32 checksum value for chunk [0, 0x35f85): Mirror 1: 0xce4acdb3 Mirror 32: 0xce4acdb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f85: skipped 000987 1763876629.279454 read 0x035f53 thru 0x035f84 (0x32 bytes) 000988 1763876629.282517 read 0x01d2df thru 0x02c7c3 (0xf4e5 bytes) 000989 1763876629.287439 write 0x03e852 thru 0x03ffff (0x17ae bytes) 000990 1763876629.299101 write 0x00c9bd thru 0x015df7 (0x943b bytes) 000991 1763876629.302541 mapwrite 0x035596 thru 0x037131 (0x1b9c bytes) 000992 1763876629.308941 mapread 0x020952 thru 0x02ed74 (0xe423 bytes) 000993 1763876629.309854 fallocate 0x03a8be thru 0x03f741 (0x4e84 bytes) 000994 1763876629.311702 fallocate 0x03e7dc thru 0x03ed12 (0x537 bytes) 000995 1763876629.313372 trunc from 0x040000 to 0x02873e (0x178c1 bytes) 000996 1763876629.323374 write 0x02a4b2 thru 0x038f61 (0xeab0 bytes) 000997 1763876629.324427 write 0x01c33d thru 0x024185 (0x7e49 bytes) 000998 1763876629.325045 read 0x02bf7b thru 0x033f6e (0x7ff4 bytes) 000999 1763876629.325395 trunc from 0x038f62 to 0x02399c (0x155c5 bytes) 001000 1763876629.330570 write 0x01efa0 thru 0x02589d (0x68fe bytes) All operations completed A-OK! PASS 70a (30s) == sanity-flr test 71: check mirror extend parent fid ==== 00:43:50 (1763876630) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre [0x240000404:0x7e5:0x0] /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 200a: stress test ===================== 00:43:54 (1763876634) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4971840 .. Extending file size to 1727840 .. done Extending file size to 9129440 .. Extending file size to 8971328 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9288480 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63998 63999 64000 64002 64005 failed 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:44:59 (1763876699) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1656768 .. Extending file size to 2473920 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 2058784 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7544672 .. Extending file size to 7123008 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8066976 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8589152 .. Extending file size to 8139424 .. done Extending file size to 8863968 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8720352 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8894368 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8987648 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8964832 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9178336 .. Waiting 68254 68255 68257 68259 68261 failed 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:46:04 (1763876764) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2920 1268384 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2952 1268352 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19948 3587072 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17600 3589420 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37548 7176492 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2920 1268384 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2952 1268352 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 19948 3587072 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17600 3589420 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 37548 7176492 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0377795 s, 55.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0516554 s, 40.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe3c5769c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:46:10 (1763876770) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0289137 s, 72.5 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 00:46:13 (1763876773) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0xce:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:46:15 (1763876775) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0230104 s, 91.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:46:18 (1763876778) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.022766 s, 92.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd2:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:46:20 (1763876780) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0238772 s, 87.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd4:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:46:44 (1763876804) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176446 s, 59.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 204d (36s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:47:20 (1763876840) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0232722 s, 90.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (35s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:47:55 (1763876875) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg413-server: Pool lustre.test_204f created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0218287 s, 96.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc06:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc06:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xc06:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc06:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg413-server: Pool lustre.test_204f destroyed PASS 204f (49s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:48:44 (1763876924) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:48:47 (1763876927) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:48:49 (1763876929) oleg413-server: Pool lustre.test_206 created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg413-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:49:04 (1763876944) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315714 s, 33.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136655 s, 76.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x68b9e92b Mirror 3: 0x68b9e92b PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:49:07 (1763876947) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:49:08 (1763876948) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:49:10 (1763876950) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0103099 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa30000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xa90000 Read this (4 bytes): BBBB 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:49:26 (1763876966) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00372199 s, 2.2 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:49:28 (1763876968) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167119 s, 62.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbe lmm_fid: [0x200000405:0xbe:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbe lmm_fid: [0x200000405:0xbe:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611326 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-23 00:49:29.000000000 -0500 Modify: 2025-11-23 00:49:29.000000000 -0500 Change: 2025-11-23 00:49:29.000000000 -0500 Birth: 2025-11-23 00:49:28.000000000 -0500 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:49:31 (1763876971) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148917 s, 70.4 MB/s oleg413-server: debugfs 1.47.3-wc1 (8-Aug-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg413-server: debugfs 1.47.3-wc1 (8-Aug-2025) oleg413-server: O/280000401/d4/e4: File not found by ext2_lookup PASS 210b (13s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:49:44 (1763876984) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0241643 s, 1.7 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00437456 s, 936 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 23 00:49 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc3:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 23 00:49 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:49:47 (1763876987) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:49:49 (1763876989) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 821 sec ============ 00:49:52 (1763876992) === sanity-flr: start cleanup 00:49:52 (1763876992) === === sanity-flr: finish cleanup 00:49:54 (1763876994) ===