-----============= acceptance-small: sanity-flr ============----- Wed Aug 12 15:34:44 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg458-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:35:06 (1786563306) === oleg458-client.virtnet: executing check_config_client /mnt/lustre oleg458-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg458-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d6a488ff800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d6a488ff800.idle_timeout=debug disable quota as required oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:35:50 (1786563350) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8241] == sanity-flr test 0a: lfs mirror create with -N option == 15:35:52 (1786563352) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:36:09 (1786563369) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:36:12 (1786563372) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:36:16 (1786563376) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:36:41 (1786563401) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:36:43 (1786563403) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 15:36:46 (1786563406) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d6a488ff800.stats=clear osc.lustre-OST0001-osc-ffff8d6a488ff800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:36:54 (1786563414) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:37:05 (1786563425) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:37:13 (1786563433) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:37:16 (1786563436) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:37:24 (1786563444) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 15:37:33 (1786563453) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:37:41 (1786563461) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:37:51 (1786563471) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:37:59 (1786563479) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:38:10 (1786563490) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:38:16 (1786563496) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.46005 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952023 s, 11.0 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:38:25 (1786563505) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325093 s, 32.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:38:33 (1786563513) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:38:45 (1786563525) Stopping /mnt/lustre-ost1 (opts:) on oleg458-server oleg458-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d6a488ff800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d6a488ff800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg458-server: oleg458-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg458-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg458-server oleg458-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d6a488ff800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d6a488ff800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg458-server: oleg458-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg458-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (78s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:40:03 (1786563603) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.425185 s, 14.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg458-server oleg458-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d6a488ff800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d6a488ff800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg458-server: oleg458-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg458-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:40:51 (1786563651) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg458-server oleg458-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d6a488ff800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d6a488ff800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg458-server: oleg458-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg458-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg458-server oleg458-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d6a488ff800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d6a488ff800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg458-server: oleg458-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg458-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:42:17 (1786563737) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.82133 s, 18.6 MB/s -rw-r--r-- 1 root root 52428800 Aug 12 15:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.076s user 0m0.000s sys 0m0.190s Stopping /mnt/lustre-ost1 (opts:) on oleg458-server oleg458-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d6a488ff800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d6a488ff800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.014s user 0m0.000s sys 0m0.179s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg458-server: oleg458-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg458-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg458-server oleg458-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d6a488ff800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d6a488ff800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.251s user 0m0.002s sys 0m0.203s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg458-server: oleg458-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg458-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg458-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d6a488ff800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d6a488ff800.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:43:55 (1786563835) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:43:58 (1786563838) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:44:02 (1786563842) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 15:44:06 (1786563846) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 15:44:16 (1786563856) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.696859 s, 18.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:44:31 (1786563871) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309689 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305359 s, 34.3 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:44:39 (1786563879) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:44:48 (1786563888) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000961573 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00148765 s, 688 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00249157 s, 411 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101502 s, 1.0 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 15:44:59 (1786563899) create test files with size 15741512 .. e4c741a11bb5e6829f40deac8dae335c - 92d9cc772f225f131c82572f5986cd3c - ec0343c7b650c9f8982c5cefe3e212aa - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg458-client.virtnet /mnt/lustre (opts:) Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 15:45:25 (1786563925) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:45:38 (1786563938) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:45:45 (1786563945) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592681 s, 17.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292178 s, 35.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 15:45:53 (1786563953) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0882223 s, 23.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.337017 s, 12.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205818 s, 10.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 15:46:12 (1786563972) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 15:46:16 (1786563976) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:46:19 (1786563979) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496732 s, 21.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 15:46:28 (1786563988) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.444353 s, 7.1 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:46:39 (1786563999) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:46:50 (1786564010) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.360701 s, 29.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg458-client.virtnet /mnt/lustre (opts:) Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:47:02 (1786564022) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8d6a53298000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.197958 s, 53.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg458-client.virtnet /mnt/lustre (opts:) Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8d6a5d6e4800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:47:18 (1786564038) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:48:01 (1786564081) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:48:42 (1786564122) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 15:48:46 (1786564126) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.29377 s, 10.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.236646 s, 13.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf79bd927 (nosync,stale) Mirror 2: 0xf79bd927 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 771b0b257026eb6b723f681749443173 - mirror 1 checksum: ae2016768563c8cef488709bc699869c - mirror 2 checksum: 771b0b257026eb6b723f681749443173 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T15:48:49-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-12T15:48:49-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 771b0b257026eb6b723f681749443173 - mirror 1 checksum: ae2016768563c8cef488709bc699869c - mirror 2 checksum: ae2016768563c8cef488709bc699869c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T15:48:49-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-12T15:48:49-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:48:58 (1786564138) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431497 s, 24.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:49:07 (1786564147) Failing mds1 on oleg458-server Stopping /mnt/lustre-mds1 (opts:) on oleg458-server 15:49:20 (1786564160) shut down facet: mds1 facet_host: oleg458-server facet_failover_host: oleg458-server Failover mds1 to oleg458-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg458-server: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-MDT0000 15:49:53 (1786564193) targets are mounted 15:49:53 (1786564193) facet_failover done oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446101 s, 23.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:50:14 (1786564214) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0758576 s, 270 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0349986 s, 585 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (14s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:50:28 (1786564228) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.104926 s, 195 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0114709 s, 1.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:50:40 (1786564240) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0154652 s, 1.1 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:50:49 (1786564249) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0608704 s, 336 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:50:59 (1786564259) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00774372 s, 529 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0237901 s, 172 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:51:09 (1786564269) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (29s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:51:39 (1786564299) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:52:09 (1786564329) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:52:35 (1786564355) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:53:01 (1786564381) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (12s) == sanity-flr test 70: mirror create and split race ====== 15:53:13 (1786564393) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57591 mirror split pid 57592 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57591 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57592 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 15:54:22 (1786564462) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 4464 skipping zero length zero range fallocating to largest ever: 0x29814 000004 1786564465.082842 fallocate 0x01f48c thru 0x029813 (0xa388 bytes) 000005 1786564465.088890 trunc from 00000000 to 0x00854a (0x854b bytes) 000006 1786564465.105960 fallocate 0x0182f5 thru 0x01e2c1 (0x5fcd bytes) 000007 1786564465.112219 mapwrite 0x035c34 thru 0x03ffff (0xa3cc bytes) 000008 1786564465.211990 mapwrite 0x03de2c thru 0x03ffff (0x21d4 bytes) truncating to largest ever: 0x39e52 000010 1786564465.255421 trunc from 0x040000 to 0x039e51 (0x61ae bytes) 000011 1786564465.287176 trunc from 0x039e52 to 0x00a5ca (0x2f887 bytes) 000012 1786564465.320983 write 0x03c029 thru 0x03c113 (0xeb bytes) 000013 1786564465.322179 read 0x007de7 thru 0x013f3e (0xc158 bytes) 000015 1786564465.341523 read 0x0245d6 thru 0x026fd3 (0x29fe bytes) 000016 1786564465.363675 read 0x03a5d0 thru 0x03c113 (0x1b44 bytes) 000017 1786564465.387159 MIRROR_EXTEND to 2 mirrors 000018 1786564466.201893 mapwrite 0x017a0c thru 0x01ee64 (0x7459 bytes) 000019 1786564466.328352 write 0x03c111 thru 0x03e1d1 (0x20c1 bytes) 000020 1786564466.343793 mapread 0x02ced2 thru 0x033daa (0x6ed9 bytes) 000021 1786564466.357705 write 0x01856d thru 0x01e106 (0x5b9a bytes) 000022 1786564466.360896 fallocate 0x016f79 thru 0x02614c (0xf1d4 bytes) 000023 1786564466.367378 punch from 0x008d8e to 0x00ccd8 (0x3f4b bytes) 000024 1786564466.373187 trunc from 0x03e1d2 to 0x01e7e0 (0x1f9f1 bytes) 000025 1786564466.450385 mapwrite 0x03c376 thru 0x03ffff (0x3c8a bytes) 000026 1786564466.499752 punch from 0x01714b to 0x01c5b1 (0x5467 bytes) 000027 1786564466.505700 fallocate 0x0112aa thru 0x01d5cb (0xc322 bytes) truncating to largest ever: 0x3e32a 000028 1786564466.516613 trunc from 0x040000 to 0x03e329 (0x1cd6 bytes) 000029 1786564466.557137 punch from 0x02a248 to 0x02b19b (0xf54 bytes) 000030 1786564466.565205 trunc from 0x0296dc to 0x02a3c9 (0xcee bytes) 000031 1786564466.594184 MIRROR_RESYNC 2 mirrors 000032 1786564467.642446 write 0x03a4db thru 0x03d0ed (0x2c13 bytes) 000036 1786564467.767958 trunc from 0x03e32a to 0x03328d (0xb09c bytes) 000037 1786564467.820653 mapwrite 0x014304 thru 0x01e44e (0xa14b bytes) 000038 1786564467.913964 punch from 0x00b694 to 0x01849f (0xce0c bytes) 000039 1786564467.921302 mapwrite 0x021904 thru 0x023d80 (0x247d bytes) 000040 1786564468.005822 write 0x025011 thru 0x033537 (0xe527 bytes) 000044 1786564468.037915 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33538): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33538): Mirror 2: 0x83b926ba (stale) CRC-32 checksum value for chunk [0, 0x33538): Mirror 1: 0xfd979616 Mirror 3: 0xfd979616 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33538: skipped 000045 1786564469.649849 punch from 0x0316c1 to 0x033537 (0x1e77 bytes) 000046 1786564469.757757 mapread 0x019180 thru 0x023de2 (0xac63 bytes) 000047 1786564469.812271 mapread 0x002e26 thru 0x008996 (0x5b71 bytes) 000048 1786564469.848276 trunc from 0x033538 to 0x029414 (0xa123 bytes) 000049 1786564469.894299 punch from 0x0129fc to 0x01f48d (0xca92 bytes) 000050 1786564469.901569 read 0x0220b4 thru 0x02247b (0x3c8 bytes) 000051 1786564469.902593 trunc from 0x029415 to 0x01fe46 (0x95ce bytes) 000052 1786564469.923214 write 0x010f95 thru 0x017828 (0x6894 bytes) 000053 1786564469.938175 MIRROR_RESYNC 3 mirrors 000055 1786564470.850655 MIRROR_EXTEND to 4 mirrors 000056 1786564471.764892 trunc from 0x018d45 to 0x01fe46 (0x7102 bytes) 000057 1786564471.851562 trunc from 0x01edb5 to 0x01fe46 (0x1092 bytes) 000058 1786564471.867051 read 0x01acb4 thru 0x01fe46 (0x5193 bytes) 000059 1786564471.886695 trunc from 0x01fe47 to 0x0016ed (0x1e759 bytes) 000060 1786564471.918862 mapwrite 0x0330b7 thru 0x0339ca (0x914 bytes) 000061 1786564471.960959 read 0x01113f thru 0x01a03e (0x8f00 bytes) 000063 1786564471.978189 read 0x0066ef thru 0x009cf8 (0x360a bytes) 000064 1786564471.979301 trunc from 0x0339cb to 0x0085ae (0x2b41c bytes) 000065 1786564472.000885 mapread 0x002fee thru 0x0085ae (0x55c1 bytes) 000066 1786564472.002619 mapread 0x0028e5 thru 0x0085ae (0x5cca bytes) 000067 1786564472.005019 trunc from 0x0085af to 0x032acd (0x2a51f bytes) 000068 1786564472.026564 trunc from 0x014919 to 0x01e7b5 (0x9e9d bytes) 000070 1786564472.035233 write 0x007eb0 thru 0x00a3c7 (0x2518 bytes) 000071 1786564472.050591 MIRROR_RESYNC 4 mirrors 000072 1786564473.168688 write 0x026f1d thru 0x02cf8d (0x6071 bytes) 000073 1786564473.240590 trunc from 0x0137a2 to 0x01e7c1 (0xb020 bytes) 000074 1786564473.247792 punch from 0x02456b to 0x02b8ff (0x7395 bytes) 000075 1786564473.304225 trunc from 0x03084e to 0x032acd (0x2280 bytes) 000076 1786564473.324141 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32ace): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ace): Mirror 2: 0xf49910d5 (stale) Mirror 3: 0xf49910d5 (stale) Mirror 4: 0xf49910d5 (stale) CRC-32 checksum value for chunk [0, 0x32ace): Mirror 1: 0x3e66c248 Mirror 5: 0x3e66c248 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ace: skipped 000077 1786564474.601946 trunc from 0x0213bf to 0x02f999 (0xe5db bytes) 000078 1786564474.666483 mapwrite 0x03b959 thru 0x03ffff (0x46a7 bytes) 000079 1786564474.810879 read 0x014b08 thru 0x0206a6 (0xbb9f bytes) 000080 1786564474.841755 trunc from 0x03fa32 to 0x03ffff (0x5ce bytes) 000081 1786564474.850769 write 0x00abf5 thru 0x018b7b (0xdf87 bytes) 000082 1786564474.888794 MIRROR_RESYNC 5 mirrors 000083 1786564476.095995 mapwrite 0x02a434 thru 0x02c7fe (0x23cb bytes) 000084 1786564476.218753 mapwrite 0x02d10a thru 0x03b8e4 (0xe7db bytes) 000087 1786564476.313187 mapread 0x0029ea thru 0x00a7ee (0x7e05 bytes) 000088 1786564476.340519 mapread 0x02198c thru 0x02e764 (0xcdd9 bytes) 000090 1786564476.369775 mapwrite 0x027a13 thru 0x02c889 (0x4e77 bytes) 000091 1786564476.411189 trunc from 0x009760 to 0x0151e0 (0xba81 bytes) 000092 1786564476.424065 trunc from 0x012e01 to 0x013832 (0xa32 bytes) 000093 1786564476.435164 fallocate 0x02491f thru 0x0297ad (0x4e8f bytes) 000094 1786564476.445162 punch from 0x009bbb to 0x017957 (0xdd9d bytes) 000097 1786564476.455120 read 0x0270b1 thru 0x0333b9 (0xc309 bytes) 000098 1786564476.456180 trunc from 0x0062cb to 0x00c6f9 (0x642f bytes) 000100 1786564476.472134 trunc from 0x040000 to 0x007b72 (0x3848d bytes) 000101 1786564476.506166 mapread 0x004cd9 thru 0x007b72 (0x2e9a bytes) 000103 1786564476.515793 punch from 0x005204 to 0x007b72 (0x296f bytes) 000104 1786564476.521869 fallocate 0x01ab31 thru 0x029350 (0xe820 bytes) 000107 1786564476.529179 mapread 0x000e87 thru 0x005207 (0x4381 bytes) 000108 1786564476.542254 punch from 0x01e322 to 0x029350 (0xb02f bytes) fallocating to largest ever: 0x40000 000109 1786564476.558105 fallocate 0x03a77c thru 0x03ffff (0x5884 bytes) 000110 1786564476.579327 trunc from 0x033dac to 0x03695e (0x2bb3 bytes) 000111 1786564476.603761 fallocate 0x00de7b thru 0x0158b6 (0x7a3c bytes) 000112 1786564476.636366 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5df56b08 (stale) Mirror 4: 0x5df56b08 (stale) Mirror 5: 0x5df56b08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5df56b08 (stale) Mirror 4: 0x5df56b08 (stale) Mirror 5: 0x5df56b08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000113 1786564477.355191 mapread 0x02cd20 thru 0x037d27 (0xb008 bytes) 000114 1786564477.399316 write 0x036b96 thru 0x03a0c3 (0x352e bytes) 000115 1786564477.408984 read 0x0051a4 thru 0x00b3ef (0x624c bytes) 000116 1786564477.423095 read 0x02007b thru 0x02047b (0x401 bytes) 000117 1786564477.440068 fallocate 0x01adad thru 0x02a013 (0xf267 bytes) 000118 1786564477.453893 MIRROR_RESYNC 4 mirrors 000119 1786564478.869767 fallocate 0x00bbc0 thru 0x00be0c (0x24d bytes) 000120 1786564478.942331 read 0x015c7d thru 0x01c976 (0x6cfa bytes) 000122 1786564478.971515 mapread 0x003fb7 thru 0x0071e9 (0x3233 bytes) 000123 1786564479.032502 write 0x012aad thru 0x01a302 (0x7856 bytes) 000124 1786564479.055980 trunc from 0x040000 to 0x02384e (0x1c7b1 bytes) 000125 1786564479.112228 mapread 0x01d85c thru 0x02384e (0x5ff3 bytes) 000126 1786564479.130261 trunc from 0x02384f to 0x029f56 (0x6708 bytes) 000127 1786564479.173070 fallocate 0x0394ac thru 0x03ffff (0x6b54 bytes) 000128 1786564479.184319 read 0x029eb9 thru 0x038f1f (0xf067 bytes) 000129 1786564479.201921 fallocate 0x01a2da thru 0x01c885 (0x25ac bytes) 000130 1786564479.215960 mapwrite 0x00e81d thru 0x0155b7 (0x6d9b bytes) 000131 1786564479.274470 MIRROR_RESYNC 4 mirrors 000132 1786564480.667730 MIRROR_EXTEND to 5 mirrors 000133 1786564481.582986 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) 000134 1786564482.082023 trunc from 0x0057ee to 0x00eaff (0x9312 bytes) 000136 1786564482.223594 punch from 0x006e9a to 0x00d77f (0x68e6 bytes) 000137 1786564482.235364 mapwrite 0x01bb88 thru 0x02780a (0xbc83 bytes) 000139 1786564482.323320 mapwrite 0x01a1d4 thru 0x0263fd (0xc22a bytes) 000140 1786564482.356626 punch from 0x01cb36 to 0x02abad (0xe078 bytes) 000141 1786564482.374569 mapread 0x014afd thru 0x0206f6 (0xbbfa bytes) 000142 1786564482.437021 fallocate 0x02781e thru 0x02ec05 (0x73e8 bytes) 000143 1786564482.446264 mapread 0x038363 thru 0x03ffff (0x7c9d bytes) 000144 1786564482.491868 mapread 0x032782 thru 0x034b3d (0x23bc bytes) 000145 1786564482.496008 fallocate 0x017f9a thru 0x026998 (0xe9ff bytes) 000146 1786564482.502920 mapread 0x018d48 thru 0x01b939 (0x2bf2 bytes) 000147 1786564482.509786 write 0x020aec thru 0x025b86 (0x509b bytes) 000148 1786564482.523089 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4aeae61a (stale) Mirror 5: 0x4aeae61a (stale) Mirror 6: 0x4aeae61a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2335630a Mirror 7: 0x2335630a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000149 1786564484.062301 mapwrite 0x0286d0 thru 0x030c2c (0x855d bytes) 000150 1786564484.294962 mapwrite 0x012392 thru 0x01592c (0x359b bytes) 000151 1786564484.355019 trunc from 0x0022ba to 0x009a8a (0x77d1 bytes) 000152 1786564484.359940 trunc from 0x040000 to 0x014c44 (0x2b3bb bytes) 000153 1786564484.383227 punch from 0x00c7ca to 0x014c44 (0x847b bytes) 000154 1786564484.389684 mapread 0x010c83 thru 0x014c44 (0x3fc2 bytes) 000155 1786564484.413091 mapwrite 0x015f77 thru 0x01ed12 (0x8d9c bytes) 000156 1786564484.559167 fallocate 0x018b74 thru 0x01c314 (0x37a1 bytes) 000157 1786564484.574861 trunc from 0x01ed13 to 0x02aea3 (0xc191 bytes) 000158 1786564484.613881 punch from 0x01fc14 to 0x0260a2 (0x648f bytes) 000159 1786564484.632881 mapwrite 0x029e8c thru 0x032edd (0x9052 bytes) 000160 1786564484.799794 trunc from 0x0258ff to 0x02b622 (0x5d24 bytes) 000161 1786564484.811893 trunc from 0x02a5ac to 0x032b63 (0x85b8 bytes) 000162 1786564484.848518 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32ede): Mirror 5: 0xb33ea3d4 (stale) Mirror 6: 0xb33ea3d4 (stale) Mirror 7: 0xce7a507b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ede): Mirror 5: 0xb33ea3d4 (stale) Mirror 6: 0xb33ea3d4 (stale) Mirror 7: 0xce7a507b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ede: skipped 000163 1786564485.545231 mapread 0x00faf9 thru 0x01bc80 (0xc188 bytes) 000164 1786564485.587176 fallocate 0x02a08c thru 0x031e67 (0x7ddc bytes) 000165 1786564485.593573 mapread 0x00d6e6 thru 0x015d25 (0x8640 bytes) 000166 1786564485.596323 write 0x032021 thru 0x03c905 (0xa8e5 bytes) 000167 1786564485.616705 trunc from 0x00f779 to 0x01b090 (0xb918 bytes) 000169 1786564485.622202 fallocate 0x00038e thru 0x0043ce (0x4041 bytes) 000171 1786564485.636265 mapread 0x018efe thru 0x01a9b4 (0x1ab7 bytes) 000172 1786564485.654915 trunc from 0x03c906 to 0x03127c (0xb689 bytes) 000173 1786564485.678724 trunc from 0x03127d to 0x011a5c (0x1f820 bytes) 000174 1786564485.702287 mapread 0x00071f thru 0x00d235 (0xcb17 bytes) 000175 1786564485.720265 mapread 0x011820 thru 0x011a5c (0x23d bytes) 000176 1786564485.722864 mapwrite 0x0177a3 thru 0x0188de (0x113c bytes) 000177 1786564485.761702 write 0x038743 thru 0x03aeeb (0x27a9 bytes) 000178 1786564485.764368 trunc from 0x01f136 to 0x02d9e7 (0xe8b2 bytes) 000179 1786564485.770743 mapread 0x03244c thru 0x0338de (0x1493 bytes) 000180 1786564485.780345 read 0x0163c9 thru 0x019c83 (0x38bb bytes) 000181 1786564485.786585 mapwrite 0x00e61e thru 0x017b08 (0x94eb bytes) 000182 1786564485.808659 write 0x02748e thru 0x02b9c1 (0x4534 bytes) 000183 1786564485.815782 trunc from 0x021e6b to 0x027fc1 (0x6157 bytes) 000184 1786564485.829866 write 0x02cdcb thru 0x031eea (0x5120 bytes) 000186 1786564485.833777 read 0x030e94 thru 0x03aeeb (0xa058 bytes) 000187 1786564485.835145 mapread 0x016c41 thru 0x018249 (0x1609 bytes) 000188 1786564485.840357 write 0x015034 thru 0x0184fd (0x34ca bytes) 000189 1786564485.847607 punch from 0x00ce5d to 0x017a79 (0xac1d bytes) 000191 1786564485.865959 read 0x0174c0 thru 0x01b570 (0x40b1 bytes) 000192 1786564485.880803 fallocate 0x00e876 thru 0x01a22d (0xb9b8 bytes) 000194 1786564485.887675 punch from 0x00103f to 0x00281c (0x17de bytes) 000195 1786564485.893506 mapread 0x02a77b thru 0x038bad (0xe433 bytes) 000196 1786564485.900088 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3aeec): Mirror 5: 0x21e6a47f (stale) Mirror 7: 0x2b2b546f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aeec): Mirror 5: 0x21e6a47f (stale) Mirror 7: 0x2b2b546f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aeec: skipped 000197 1786564486.449889 read 0x00afef thru 0x00e898 (0x38aa bytes) 000198 1786564486.470631 fallocate 0x014a44 thru 0x015fb2 (0x156f bytes) 000200 1786564486.481888 fallocate 0x03b258 thru 0x03f546 (0x42ef bytes) 000202 1786564486.491882 punch from 0x01c31c to 0x021184 (0x4e69 bytes) 000205 1786564486.500623 read 0x00f896 thru 0x0109c3 (0x112e bytes) 000206 1786564486.519194 fallocate 0x011825 thru 0x012b8a (0x1366 bytes) 000207 1786564486.530932 write 0x01a7dd thru 0x025406 (0xac2a bytes) 000208 1786564486.541615 punch from 0x018a0a to 0x02857d (0xfb74 bytes) 000209 1786564486.566726 punch from 0x030c8e to 0x033d16 (0x3089 bytes) 000210 1786564486.572809 write 0x03b3d9 thru 0x03ffff (0x4c27 bytes) 000211 1786564486.574477 trunc from 0x040000 to 0x01f84d (0x207b2 bytes) 000212 1786564486.611777 mapread 0x003f01 thru 0x012233 (0xe333 bytes) 000213 1786564486.618730 punch from 0x007045 to 0x00bc72 (0x4c2e bytes) 000214 1786564486.628224 punch from 0x00d93f to 0x00da15 (0xd7 bytes) 000215 1786564486.634694 trunc from 0x000676 to 0x00a36e (0x9cf9 bytes) 000216 1786564486.641782 write 0x02e2eb thru 0x03dacb (0xf7e1 bytes) 000217 1786564486.645570 write 0x02af97 thru 0x0345ca (0x9634 bytes) 000218 1786564486.653790 punch from 0x01059e to 0x010e64 (0x8c7 bytes) 000219 1786564486.660026 mapwrite 0x003262 thru 0x00b70c (0x84ab bytes) 000220 1786564486.732898 write 0x0339a0 thru 0x03ffff (0xc660 bytes) 000221 1786564486.735295 punch from 0x011445 to 0x01cece (0xba8a bytes) 000222 1786564486.745697 write 0x004cf7 thru 0x00cc92 (0x7f9c bytes) 000223 1786564486.748912 mapwrite 0x002e72 thru 0x004c97 (0x1e26 bytes) 000224 1786564486.776735 punch from 0x023d3f to 0x02e867 (0xab29 bytes) 000225 1786564486.782959 mapwrite 0x01bc10 thru 0x01c543 (0x934 bytes) 000226 1786564486.815339 MIRROR_RESYNC 3 mirrors 000227 1786564488.291556 read 0x01b624 thru 0x01f597 (0x3f74 bytes) 000228 1786564488.316765 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) 000230 1786564488.807068 trunc from 0x040000 to 0x005241 (0x3adbe bytes) 000231 1786564488.894846 fallocate 0x02efcd thru 0x036f74 (0x7fa8 bytes) 000232 1786564488.913458 trunc from 0x002237 to 0x005241 (0x300b bytes) 000233 1786564488.922811 mapread 0x000b72 thru 0x002bb5 (0x2044 bytes) 000234 1786564488.937422 read 0x0036b8 thru 0x0051f2 (0x1b3b bytes) 000235 1786564488.944109 read 0x004e96 thru 0x005241 (0x3ac bytes) 000236 1786564488.945080 read 0x00319d thru 0x005241 (0x20a5 bytes) 000237 1786564488.945928 trunc from 0x004719 to 0x005241 (0xb29 bytes) 000238 1786564488.951058 trunc from 0x0006e2 to 0x004661 (0x3f80 bytes) 000239 1786564488.967378 mapread 0x003a4a thru 0x005241 (0x17f8 bytes) 000241 1786564488.991644 trunc from 0x00188c to 0x005241 (0x39b6 bytes) 000242 1786564489.000947 mapread 0x004c6d thru 0x005241 (0x5d5 bytes) 000243 1786564489.029960 punch from 0x0014f4 to 0x005241 (0x3d4e bytes) 000246 1786564489.045792 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x5242): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5242): Mirror 7: 0x729b40ac (stale) CRC-32 checksum value for chunk [0, 0x5242): Mirror 1: 0x472cab8c Mirror 8: 0x472cab8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5242: skipped 000247 1786564490.208622 mapread 0x00117f thru 0x005241 (0x40c3 bytes) 000248 1786564490.276204 mapread 0x0013af thru 0x005241 (0x3e93 bytes) 000249 1786564490.311593 fallocate 0x03f5e9 thru 0x03ffff (0xa17 bytes) 000250 1786564490.386184 trunc from 0x040000 to 0x015292 (0x2ad6d bytes) 000251 1786564490.413472 punch from 0x00001c to 0x009c7b (0x9c60 bytes) 000252 1786564490.419598 trunc from 0x00f8ea to 0x015292 (0x59a9 bytes) 000253 1786564490.424837 fallocate 0x03ecda thru 0x03ffff (0x1326 bytes) 000254 1786564490.436305 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000256 1786564490.960845 punch from 0x029d46 to 0x0357f5 (0xbab0 bytes) 000257 1786564490.976523 trunc from 0x02fea3 to 0x0362c2 (0x6420 bytes) 000258 1786564490.984606 trunc from 0x040000 to 0x0109ba (0x2f645 bytes) 000259 1786564491.043625 MIRROR_RESYNC 2 mirrors 000260 1786564492.038357 mapwrite 0x025e1b thru 0x0316b0 (0xb896 bytes) 000261 1786564492.253738 fallocate 0x028b0a thru 0x033218 (0xa70f bytes) 000262 1786564492.261888 write 0x007fed thru 0x010a57 (0x8a6b bytes) 000263 1786564492.277364 punch from 0x016368 to 0x020be8 (0xa881 bytes) 000264 1786564492.284337 punch from 0x01e103 to 0x029673 (0xb571 bytes) 000265 1786564492.292129 write 0x03560a thru 0x03d016 (0x7a0d bytes) 000266 1786564492.294796 trunc from 0x03d017 to 0x020407 (0x1cc0f bytes) 000267 1786564492.338755 fallocate 0x00def0 thru 0x0131d3 (0x52e4 bytes) 000268 1786564492.349512 mapwrite 0x02630c thru 0x02c27e (0x5f73 bytes) 000269 1786564492.448788 trunc from 0x02c27f to 0x0111bb (0x1b0c3 bytes) 000270 1786564492.490616 fallocate 0x0224c4 thru 0x02a5a1 (0x80de bytes) 000273 1786564492.501988 mapread 0x008367 thru 0x00ddca (0x5a64 bytes) 000274 1786564492.504002 read 0x004cfe thru 0x0111bb (0xc4be bytes) 000275 1786564492.522497 trunc from 0x004e57 to 0x009b03 (0x4cad bytes) 000276 1786564492.537819 trunc from 0x0111bc to 0x02e42a (0x1d26f bytes) 000277 1786564492.591665 mapwrite 0x029c72 thru 0x035d73 (0xc102 bytes) 000278 1786564492.688009 mapread 0x02666a thru 0x02e773 (0x810a bytes) 000279 1786564492.690680 trunc from 0x0010be to 0x00b38f (0xa2d2 bytes) 000280 1786564492.697833 trunc from 0x035d74 to 0x0356a1 (0x6d2 bytes) 000281 1786564492.727694 trunc from 0x0356a2 to 0x025c1a (0xfa87 bytes) 000282 1786564492.751034 punch from 0x00a38a to 0x017966 (0xd5dd bytes) 000283 1786564492.758669 mapwrite 0x022151 thru 0x028628 (0x64d8 bytes) 000284 1786564492.816058 trunc from 0x00dfb2 to 0x01d8c5 (0xf914 bytes) 000285 1786564492.829982 mapread 0x020c23 thru 0x028628 (0x7a06 bytes) 000286 1786564492.846522 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) 000287 1786564493.065559 write 0x037a8a thru 0x03a73b (0x2cb2 bytes) 000288 1786564493.074000 trunc from 0x03a73c to 0x01bcec (0x1ea4f bytes) 000289 1786564493.096856 write 0x011ad0 thru 0x01672f (0x4c60 bytes) 000290 1786564493.108882 read 0x00070b thru 0x005203 (0x4af9 bytes) 000292 1786564493.119782 punch from 0x00c407 to 0x00f1d9 (0x2dd3 bytes) 000293 1786564493.130929 MIRROR_EXTEND to 2 mirrors 000294 1786564493.781988 read 0x00f7a9 thru 0x01bcec (0xc544 bytes) 000295 1786564493.798516 fallocate 0x03db18 thru 0x03ffff (0x24e8 bytes) 000296 1786564493.856933 read 0x01747a thru 0x017bce (0x755 bytes) 000297 1786564493.881592 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) 000298 1786564494.061187 punch from 0x011d08 to 0x01bcec (0x9fe5 bytes) 000299 1786564494.070448 fallocate 0x032a53 thru 0x03ab5f (0x810d bytes) 000300 1786564494.080694 fallocate 0x03cb55 thru 0x03ffff (0x34ab bytes) 000301 1786564494.087537 mapwrite 0x0060ac thru 0x013479 (0xd3ce bytes) 000302 1786564494.169670 fallocate 0x01b8d0 thru 0x01d99e (0x20cf bytes) 000303 1786564494.176647 read 0x02917e thru 0x0364ba (0xd33d bytes) 000304 1786564494.191821 trunc from 0x013990 to 0x022dd3 (0xf444 bytes) 000305 1786564494.200457 trunc from 0x040000 to 0x033115 (0xceea bytes) 000306 1786564494.262187 write 0x0024d5 thru 0x009e95 (0x79c1 bytes) 000308 1786564494.264679 write 0x03ec50 thru 0x03ffff (0x13b0 bytes) 000309 1786564494.266571 fallocate 0x0313da thru 0x03fbdd (0xe804 bytes) 000310 1786564494.272065 trunc from 0x040000 to 0x0033c1 (0x3cc3e bytes) 000311 1786564494.316723 MIRROR_EXTEND to 2 mirrors 000312 1786564494.958044 mapread 0x001bc3 thru 0x0033c1 (0x17ff bytes) 000313 1786564494.978356 write 0x02eca2 thru 0x031ae8 (0x2e47 bytes) 000314 1786564495.019096 trunc from 0x00e572 to 0x019160 (0xabef bytes) 000315 1786564495.033661 write 0x03561c thru 0x03ba4d (0x6432 bytes) 000316 1786564495.038305 read 0x009dd9 thru 0x00e11b (0x4343 bytes) 000317 1786564495.062160 trunc from 0x02fb71 to 0x034f7a (0x540a bytes) 000318 1786564495.086520 mapread 0x01b5b1 thru 0x0280fc (0xcb4c bytes) 000319 1786564495.132669 mapread 0x03b09d thru 0x03ba4d (0x9b1 bytes) 000320 1786564495.138762 trunc from 0x03ba4e to 0x030abd (0xaf90 bytes) 000321 1786564495.203517 read 0x018235 thru 0x0188c7 (0x693 bytes) 000322 1786564495.205606 fallocate 0x0072d8 thru 0x00b552 (0x427b bytes) 000323 1786564495.213050 fallocate 0x0313d7 thru 0x03e9c5 (0xd5ef bytes) 000324 1786564495.230374 trunc from 0x030abe to 0x02e991 (0x212c bytes) 000325 1786564495.252798 mapread 0x01dc2e thru 0x020848 (0x2c1b bytes) 000326 1786564495.255137 write 0x028b23 thru 0x028de8 (0x2c6 bytes) 000327 1786564495.257800 trunc from 0x02e992 to 0x012a4d (0x1bf44 bytes) 000328 1786564495.300358 mapread 0x00d49b thru 0x012a4d (0x55b3 bytes) 000329 1786564495.314836 write 0x0246b5 thru 0x02e8b0 (0xa1fc bytes) 000330 1786564495.317813 trunc from 0x003b4e to 0x0139a5 (0xfe58 bytes) 000331 1786564495.327675 trunc from 0x02e8b1 to 0x03d793 (0xeee3 bytes) 000332 1786564495.381692 write 0x012f9e thru 0x021b6d (0xebd0 bytes) 000334 1786564495.414950 trunc from 0x02c018 to 0x03ac34 (0xec1d bytes) 000335 1786564495.449549 mapwrite 0x01992e thru 0x028e29 (0xf4fc bytes) 000336 1786564495.498534 punch from 0x019231 to 0x0270a6 (0xde76 bytes) 000337 1786564495.506616 write 0x020c2b thru 0x02533d (0x4713 bytes) 000338 1786564495.556336 MIRROR_RESYNC 2 mirrors 000339 1786564497.225719 read 0x03316c thru 0x03d793 (0xa628 bytes) 000340 1786564497.244489 fallocate 0x038538 thru 0x03ffff (0x7ac8 bytes) 000341 1786564497.290567 trunc from 0x036cd2 to 0x03d793 (0x6ac2 bytes) 000342 1786564497.297088 write 0x02fa85 thru 0x0383f1 (0x896d bytes) 000343 1786564497.319466 mapread 0x0316fb thru 0x03d793 (0xc099 bytes) 000345 1786564497.359955 write 0x00c48e thru 0x01083f (0x43b2 bytes) 000346 1786564497.380173 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) 000347 1786564497.707361 MIRROR_EXTEND to 2 mirrors 000348 1786564498.898215 mapwrite 0x019c78 thru 0x0264e5 (0xc86e bytes) 000349 1786564499.085093 trunc from 0x023e38 to 0x028ad9 (0x4ca2 bytes) 000350 1786564499.101636 read 0x01dba3 thru 0x01dcaa (0x108 bytes) 000351 1786564499.102481 punch from 0x00f29c to 0x01a2ee (0xb053 bytes) 000352 1786564499.132145 MIRROR_RESYNC 2 mirrors 000354 1786564500.714908 fallocate 0x03a59f thru 0x03b221 (0xc83 bytes) 000355 1786564500.784259 read 0x00fc83 thru 0x012bff (0x2f7d bytes) 000356 1786564500.804656 trunc from 0x0298aa to 0x031647 (0x7d9e bytes) 000357 1786564500.815156 trunc from 0x0063e1 to 0x009209 (0x2e29 bytes) 000358 1786564500.828940 mapread 0x00f200 thru 0x0193cd (0xa1ce bytes) 000359 1786564500.978590 read 0x009a50 thru 0x0143cb (0xa97c bytes) 000360 1786564501.028307 trunc from 0x014af3 to 0x016745 (0x1c53 bytes) 000361 1786564501.041708 fallocate 0x01c6a4 thru 0x02930e (0xcc6b bytes) 000362 1786564501.059877 read 0x014111 thru 0x021f3d (0xde2d bytes) 000363 1786564501.069671 mapread 0x01bf29 thru 0x01d516 (0x15ee bytes) 000364 1786564501.089357 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) 000365 1786564501.303380 fallocate 0x0254d7 thru 0x0332c9 (0xddf3 bytes) 000366 1786564501.320356 punch from 0x02ff67 to 0x03543e (0x54d8 bytes) 000367 1786564501.328631 punch from 0x01945a to 0x01ccca (0x3871 bytes) 000368 1786564501.339174 fallocate 0x01c695 thru 0x027e60 (0xb7cc bytes) 000369 1786564501.352898 read 0x03c261 thru 0x03d793 (0x1533 bytes) 000370 1786564501.363291 fallocate 0x003635 thru 0x010cb8 (0xd684 bytes) 000371 1786564501.378879 mapwrite 0x00ca27 thru 0x00de2a (0x1404 bytes) 000372 1786564501.450182 trunc from 0x033f6e to 0x03d535 (0x95c8 bytes) 000373 1786564501.460155 mapread 0x03b0df thru 0x03d793 (0x26b5 bytes) 000374 1786564501.477946 mapwrite 0x01203e thru 0x014956 (0x2919 bytes) 000375 1786564501.511579 read 0x03398f thru 0x03d793 (0x9e05 bytes) 000376 1786564501.530232 punch from 0x02abb2 to 0x0324f9 (0x7948 bytes) 000377 1786564501.538167 read 0x019937 thru 0x01e5f0 (0x4cba bytes) 000378 1786564501.540234 trunc from 0x03abde to 0x03d793 (0x2bb6 bytes) 000379 1786564501.548196 write 0x008000 thru 0x01535b (0xd35c bytes) 000380 1786564501.559872 trunc from 0x01b4ed to 0x0250b1 (0x9bc5 bytes) 000381 1786564501.567370 fallocate 0x00790b thru 0x00c92d (0x5023 bytes) 000383 1786564501.587203 mapwrite 0x034d89 thru 0x035a55 (0xccd bytes) 000384 1786564501.632533 trunc from 0x00bdcc to 0x012013 (0x6248 bytes) 000385 1786564501.643735 mapwrite 0x015ff4 thru 0x025972 (0xf97f bytes) 000386 1786564501.708557 trunc from 0x03d794 to 0x03b40b (0x2388 bytes) 000387 1786564501.727914 fallocate 0x00b3db thru 0x0173cd (0xbff3 bytes) 000388 1786564501.734002 fallocate 0x006b96 thru 0x00d9a8 (0x6e13 bytes) 000389 1786564501.739598 mapread 0x0114d1 thru 0x01d2ea (0xbe1a bytes) 000392 1786564501.743704 mapread 0x02a43b thru 0x02cf65 (0x2b2b bytes) 000393 1786564501.745497 trunc from 0x03b40c to 0x036301 (0x510a bytes) 000396 1786564501.775472 read 0x034598 thru 0x036301 (0x1d6a bytes) 000397 1786564501.781375 punch from 0x02c3df to 0x036301 (0x9f23 bytes) 000398 1786564501.787688 trunc from 0x036302 to 0x01d3ca (0x18f37 bytes) 000399 1786564501.806648 mapread 0x00d3c9 thru 0x00f66b (0x22a3 bytes) 000400 1786564501.808252 mapwrite 0x00b9e4 thru 0x01b0c1 (0xf6de bytes) 000401 1786564501.860224 MIRROR_EXTEND to 2 mirrors 000402 1786564502.983362 trunc from 0x01d3cb to 0x029dcf (0xca05 bytes) 000403 1786564503.063621 mapwrite 0x00c62c thru 0x017d7e (0xb753 bytes) 000404 1786564503.168543 fallocate 0x02a577 thru 0x031710 (0x719a bytes) 000405 1786564503.184597 fallocate 0x00ffbd thru 0x0172bf (0x7303 bytes) 000406 1786564503.205997 fallocate 0x027080 thru 0x031a07 (0xa988 bytes) 000407 1786564503.221130 trunc from 0x02fed1 to 0x031a07 (0x1b37 bytes) 000408 1786564503.240058 mapread 0x01c272 thru 0x0231ed (0x6f7c bytes) 000409 1786564503.281114 mapwrite 0x018347 thru 0x01edab (0x6a65 bytes) 000410 1786564503.332120 write 0x0017e7 thru 0x001b17 (0x331 bytes) 000411 1786564503.349870 mapwrite 0x02e33d thru 0x0310e7 (0x2dab bytes) 000414 1786564503.411129 read 0x01e6b9 thru 0x02b2e7 (0xcc2f bytes) 000415 1786564503.419506 mapwrite 0x01f049 thru 0x029b10 (0xaac8 bytes) 000416 1786564503.475147 punch from 0x002d58 to 0x010793 (0xda3c bytes) 000418 1786564503.486711 trunc from 0x031a08 to 0x000039 (0x319ce bytes) 000420 1786564503.542148 write 0x034b75 thru 0x03fa46 (0xaed2 bytes) 000421 1786564503.544614 mapread 0x00427b thru 0x00bc41 (0x79c7 bytes) 000422 1786564503.592284 mapread 0x0366fb thru 0x03fa46 (0x934c bytes) 000423 1786564503.605180 mapread 0x01a575 thru 0x02828a (0xdd16 bytes) 000425 1786564503.665748 punch from 0x0367a0 to 0x03ba80 (0x52e1 bytes) 000426 1786564503.706929 punch from 0x00b4a0 to 0x00e8ed (0x344e bytes) 000427 1786564503.723611 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fa47): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa47): Mirror 2: 0x95fba830 (stale) CRC-32 checksum value for chunk [0, 0x3fa47): Mirror 1: 0xf55a56db Mirror 3: 0xf55a56db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa47: skipped 000428 1786564505.130592 punch from 0x0259b2 to 0x027dc8 (0x2417 bytes) 000429 1786564505.170655 punch from 0x015356 to 0x02035b (0xb006 bytes) 000430 1786564505.176608 trunc from 0x03fa47 to 0x034fb9 (0xaa8d bytes) 000431 1786564505.205248 trunc from 0x023472 to 0x027298 (0x3e27 bytes) 000433 1786564505.211802 write 0x013529 thru 0x020a65 (0xd53d bytes) 000434 1786564505.227626 mapread 0x0153c1 thru 0x01cfc0 (0x7c00 bytes) 000435 1786564505.231215 trunc from 0x034fba to 0x024547 (0x10a72 bytes) 000436 1786564505.268863 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24548): Mirror 3: 0xdfca829b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24548): Mirror 3: 0xdfca829b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24548: skipped 000437 1786564505.682740 trunc from 0x024548 to 0x031534 (0xcfed bytes) 000438 1786564505.703508 trunc from 0x031535 to 0x02a931 (0x6c03 bytes) 000439 1786564505.729633 mapwrite 0x001ddf thru 0x003b3a (0x1d5c bytes) 000440 1786564505.765070 mapwrite 0x017840 thru 0x020247 (0x8a08 bytes) 000441 1786564505.805209 fallocate 0x024866 thru 0x02f1c1 (0xa95c bytes) 000442 1786564505.809710 fallocate 0x01712e thru 0x01b820 (0x46f3 bytes) 000443 1786564505.815163 mapwrite 0x029a68 thru 0x0331b2 (0x974b bytes) 000444 1786564505.890820 read 0x013af1 thru 0x015d37 (0x2247 bytes) 000445 1786564505.891685 fallocate 0x031345 thru 0x033e75 (0x2b31 bytes) 000446 1786564505.898129 trunc from 0x00857e to 0x012423 (0x9ea6 bytes) 000447 1786564505.903472 trunc from 0x0331b3 to 0x0143ce (0x1ede4 bytes) 000448 1786564505.920052 mapwrite 0x03cb16 thru 0x03ffff (0x34ea bytes) 000449 1786564505.961642 fallocate 0x03217b thru 0x03ec24 (0xcaaa bytes) 000450 1786564505.969926 fallocate 0x035e10 thru 0x03b11c (0x530d bytes) 000451 1786564505.974545 write 0x01867b thru 0x019bb3 (0x1539 bytes) 000452 1786564505.982949 read 0x0065de thru 0x01082a (0xa24d bytes) 000453 1786564505.992310 write 0x00a5c4 thru 0x018a4c (0xe489 bytes) 000454 1786564505.993915 mapwrite 0x004494 thru 0x00e50c (0xa079 bytes) 000455 1786564506.014265 trunc from 0x040000 to 0x02d0db (0x12f24 bytes) 000456 1786564506.027255 read 0x01a5e6 thru 0x022ba8 (0x85c3 bytes) 000457 1786564506.028028 mapread 0x0083c9 thru 0x00b863 (0x349b bytes) 000458 1786564506.030378 read 0x013ce4 thru 0x01ebef (0xaf0c bytes) 000459 1786564506.031244 mapwrite 0x01a6b5 thru 0x022b08 (0x8454 bytes) 000460 1786564506.048939 read 0x025ae9 thru 0x026427 (0x93f bytes) 000461 1786564506.049677 mapread 0x0287b2 thru 0x02d0db (0x492a bytes) 000462 1786564506.050696 trunc from 0x0068e5 to 0x014652 (0xdd6e bytes) 000463 1786564506.056880 trunc from 0x02d0dc to 0x00db0d (0x1f5ce bytes) 000464 1786564506.077207 mapread 0x00ab15 thru 0x00db0d (0x2ff9 bytes) 000465 1786564506.084861 trunc from 0x006781 to 0x00a0ba (0x393a bytes) 000466 1786564506.094302 trunc from 0x00db0e to 0x0219c2 (0x13eb5 bytes) 000467 1786564506.107130 fallocate 0x03caba thru 0x03ffff (0x3546 bytes) 000468 1786564506.121698 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc3ca61fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1e772507 Mirror 4: 0x1e772507 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000469 1786564506.976174 mapread 0x013915 thru 0x014b82 (0x126e bytes) 000470 1786564507.021745 trunc from 0x011175 to 0x019715 (0x85a1 bytes) 000471 1786564507.097017 mapwrite 0x000a19 thru 0x004159 (0x3741 bytes) 000472 1786564507.174655 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xc3ca61fe (stale) Mirror 4: 0x1e772507 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2c2202d5 Mirror 5: 0x2c2202d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000473 1786564508.544866 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1e772507 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x1e772507 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2c2202d5 Mirror 5: 0x2c2202d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000474 1786564509.273832 mapwrite 0x0375eb thru 0x03ffff (0x8a15 bytes) 000475 1786564509.533647 mapwrite 0x0111f8 thru 0x01c5ad (0xb3b6 bytes) 000476 1786564509.588276 punch from 0x035ee8 to 0x0363d3 (0x4ec bytes) 000477 1786564509.593078 trunc from 0x039e8e to 0x03ffff (0x6172 bytes) 000478 1786564509.604282 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2c2202d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2c2202d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000479 1786564509.952662 read 0x002d8a thru 0x011bce (0xee45 bytes) 000480 1786564509.976590 MIRROR_RESYNC 2 mirrors 000481 1786564511.031817 trunc from 0x02c78e to 0x03ab19 (0xe38c bytes) 000482 1786564511.075495 mapread 0x01f133 thru 0x01f854 (0x722 bytes) 000483 1786564511.086857 read 0x02550c thru 0x02bc01 (0x66f6 bytes) 000484 1786564511.100139 read 0x03d9e2 thru 0x03ffff (0x261e bytes) 000485 1786564511.112159 mapwrite 0x02ebfa thru 0x03d135 (0xe53c bytes) 000486 1786564511.158329 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1b22276b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x678d25d6 Mirror 6: 0x678d25d6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000487 1786564512.600576 trunc from 0x005b1d to 0x008d49 (0x322d bytes) 000488 1786564512.659464 mapread 0x035b3f thru 0x03ffff (0xa4c1 bytes) 000489 1786564512.720345 trunc from 0x03131f to 0x03ceec (0xbbce bytes) 000490 1786564512.740286 trunc from 0x040000 to 0x01a104 (0x25efb bytes) 000491 1786564512.846221 punch from 0x006912 to 0x014492 (0xdb81 bytes) 000492 1786564512.874206 mapread 0x003dae thru 0x004adb (0xd2e bytes) 000493 1786564512.891907 mapwrite 0x03c07d thru 0x03ffff (0x3f83 bytes) 000494 1786564513.001604 punch from 0x00de63 to 0x019bbd (0xbd5b bytes) 000495 1786564513.011831 mapread 0x023522 thru 0x0280c2 (0x4ba1 bytes) 000496 1786564513.026223 trunc from 0x040000 to 0x03948a (0x6b75 bytes) 000497 1786564513.054416 mapwrite 0x00bb9f thru 0x011d4a (0x61ac bytes) 000500 1786564513.091168 trunc from 0x03948b to 0x005185 (0x34305 bytes) 000501 1786564513.114749 read 0x00324a thru 0x005185 (0x1f3c bytes) 000502 1786564513.122685 MIRROR_RESYNC 3 mirrors 000503 1786564514.076862 trunc from 0x003d49 to 0x005185 (0x143d bytes) 000504 1786564514.146748 trunc from 0x005186 to 0x020f5c (0x1bdd7 bytes) 000505 1786564514.232073 fallocate 0x00cc6e thru 0x00e2f3 (0x1686 bytes) 000506 1786564514.252637 mapwrite 0x01e083 thru 0x02b24c (0xd1ca bytes) 000507 1786564514.356123 fallocate 0x01f82c thru 0x02b712 (0xbee7 bytes) 000508 1786564514.362176 fallocate 0x03b2a8 thru 0x03ffff (0x4d58 bytes) 000509 1786564514.370956 read 0x021e86 thru 0x0222bb (0x436 bytes) 000511 1786564514.373134 trunc from 0x00d8d2 to 0x01346c (0x5b9b bytes) 000512 1786564514.380753 mapwrite 0x02caa4 thru 0x035863 (0x8dc0 bytes) 000513 1786564514.482870 trunc from 0x0289a3 to 0x0339fc (0xb05a bytes) 000514 1786564514.497383 fallocate 0x026b08 thru 0x0358ea (0xede3 bytes) 000515 1786564514.508375 punch from 0x00dbd0 to 0x01d140 (0xf571 bytes) 000516 1786564514.519282 fallocate 0x015e75 thru 0x025489 (0xf615 bytes) 000518 1786564514.537230 write 0x031f81 thru 0x03f709 (0xd789 bytes) 000519 1786564514.550254 punch from 0x037f5f to 0x03e109 (0x61ab bytes) 000520 1786564514.574625 read 0x038e3b thru 0x03f709 (0x68cf bytes) 000521 1786564514.592257 read 0x015220 thru 0x0221ab (0xcf8c bytes) 000522 1786564514.600658 read 0x017056 thru 0x01b560 (0x450b bytes) 000523 1786564514.602192 write 0x02717a thru 0x0364c4 (0xf34b bytes) 000524 1786564514.606289 write 0x0136ce thru 0x019952 (0x6285 bytes) 000525 1786564514.608617 punch from 0x00fa3d to 0x011eb8 (0x247c bytes) 000526 1786564514.617355 mapwrite 0x0005a1 thru 0x00be05 (0xb865 bytes) 000527 1786564514.670508 punch from 0x0357ae to 0x038ab0 (0x3303 bytes) 000528 1786564514.683693 MIRROR_RESYNC 3 mirrors 000529 1786564516.263172 trunc from 0x03f70a to 0x037cea (0x7a1f bytes) 000530 1786564516.314683 read 0x035834 thru 0x037cea (0x24b7 bytes) 000531 1786564516.324829 read 0x028cb3 thru 0x02f40b (0x6759 bytes) 000532 1786564516.338976 mapread 0x0088ca thru 0x0175af (0xece6 bytes) 000534 1786564516.345604 mapwrite 0x002c01 thru 0x0083e8 (0x57e8 bytes) 000536 1786564516.380729 mapwrite 0x010ee8 thru 0x0125c2 (0x16db bytes) 000537 1786564516.410818 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37ceb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ceb): Mirror 5: 0xfc58f7e0 (stale) Mirror 6: 0xfc58f7e0 (stale) CRC-32 checksum value for chunk [0, 0x37ceb): Mirror 1: 0x3aa32b06 Mirror 7: 0x3aa32b06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ceb: skipped 000539 1786564518.038624 fallocate 0x004dc3 thru 0x00c080 (0x72be bytes) 000540 1786564518.133036 trunc from 0x037ceb to 0x022f14 (0x14dd6 bytes) 000541 1786564518.176894 trunc from 0x022f15 to 0x012aa8 (0x1046c bytes) 000542 1786564518.193271 punch from 0x005f70 to 0x008cf1 (0x2d82 bytes) 000544 1786564518.208579 fallocate 0x01c02f thru 0x0282c4 (0xc296 bytes) 000545 1786564518.222000 MIRROR_RESYNC 4 mirrors 000546 1786564520.261565 fallocate 0x0220ce thru 0x02998a (0x78bd bytes) 000547 1786564520.300008 mapwrite 0x009525 thru 0x00cdb5 (0x3891 bytes) 000548 1786564520.374762 write 0x025b8a thru 0x033b08 (0xdf7f bytes) 000549 1786564520.392110 mapread 0x00eec5 thru 0x0198d9 (0xaa15 bytes) 000550 1786564520.427821 mapread 0x012e2d thru 0x014ba4 (0x1d78 bytes) 000551 1786564520.440068 trunc from 0x033b09 to 0x001af9 (0x3200f bytes) 000552 1786564520.516663 mapread 0x0003f5 thru 0x001af9 (0x1705 bytes) 000553 1786564520.527325 trunc from 0x001afa to 0x00784b (0x5d52 bytes) 000554 1786564520.581341 trunc from 0x005c88 to 0x00784b (0x1bc4 bytes) 000555 1786564520.606700 trunc from 0x00784c to 0x03d04e (0x35803 bytes) 000556 1786564520.654258 trunc from 0x03d04f to 0x002da7 (0x3a2a7 bytes) 000558 1786564520.690974 trunc from 0x002da8 to 0x01a420 (0x17679 bytes) 000559 1786564520.730754 punch from 0x005590 to 0x005b4a (0x5bb bytes) 000560 1786564520.741653 trunc from 0x01a421 to 0x0385a8 (0x1e188 bytes) 000561 1786564520.768474 mapwrite 0x0133f3 thru 0x017833 (0x4441 bytes) 000562 1786564520.852996 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x385a9): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x385a9): Mirror 5: 0x2705c201 (stale) Mirror 6: 0x2705c201 (stale) Mirror 7: 0x2705c201 (stale) CRC-32 checksum value for chunk [0, 0x385a9): Mirror 1: 0xd7bfb247 Mirror 8: 0xd7bfb247 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x385a9: skipped 000563 1786564522.140595 read 0x00cd8f thru 0x01324e (0x64c0 bytes) 000564 1786564522.156493 trunc from 0x01011f to 0x016723 (0x6605 bytes) 000565 1786564522.241277 trunc from 0x009b86 to 0x013e3c (0xa2b7 bytes) 000566 1786564522.253501 trunc from 0x0385a9 to 0x0203ef (0x181b9 bytes) 000567 1786564522.284795 mapwrite 0x002616 thru 0x00d92c (0xb317 bytes) 000568 1786564522.326567 mapread 0x00231c thru 0x00abd8 (0x88bd bytes) 000569 1786564522.328014 write 0x00fb3b thru 0x01c480 (0xc946 bytes) 000570 1786564522.343486 MIRROR_RESYNC 5 mirrors 000571 1786564523.775190 MIRROR_RESYNC 5 mirrors 000572 1786564524.413835 read 0x00d3f8 thru 0x01770a (0xa313 bytes) 000573 1786564524.437243 read 0x01b915 thru 0x0203ef (0x4adb bytes) 000574 1786564524.449176 write 0x0112d7 thru 0x01ce6a (0xbb94 bytes) 000575 1786564524.517578 trunc from 0x006fbf to 0x0149ac (0xd9ee bytes) 000576 1786564524.543193 mapwrite 0x01663b thru 0x018a9a (0x2460 bytes) 000577 1786564524.560079 trunc from 0x01d6eb to 0x01dbf1 (0x507 bytes) 000578 1786564524.574941 read 0x014246 thru 0x01a8b2 (0x666d bytes) 000579 1786564524.594163 mapread 0x01acd0 thru 0x01d8e7 (0x2c18 bytes) 000580 1786564524.597041 write 0x009237 thru 0x0189d8 (0xf7a2 bytes) 000581 1786564524.600373 trunc from 0x0203f0 to 0x0034fd (0x1cef2 bytes) 000582 1786564524.660528 mapread 0x00250a thru 0x002dfc (0x8f3 bytes) 000583 1786564524.662532 write 0x00cb77 thru 0x018b41 (0xbfcb bytes) 000584 1786564524.672951 mapwrite 0x034dca thru 0x03ffff (0xb236 bytes) 000585 1786564524.782522 trunc from 0x040000 to 0x0254bd (0x1ab42 bytes) 000587 1786564524.819281 read 0x0144f9 thru 0x01edc0 (0xa8c8 bytes) 000588 1786564524.847200 trunc from 0x017c21 to 0x01d654 (0x5a34 bytes) 000589 1786564524.856509 read 0x016c3b thru 0x019285 (0x264b bytes) 000590 1786564524.870243 mapwrite 0x015fd2 thru 0x02175d (0xb78c bytes) 000591 1786564524.910087 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x254be): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x254be): Mirror 5: 0x272681cd (stale) Mirror 6: 0x272681cd (stale) Mirror 7: 0x272681cd (stale) Mirror 8: 0x272681cd (stale) CRC-32 checksum value for chunk [0, 0x254be): Mirror 1: 0x3f045291 Mirror 9: 0x3f045291 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x254be: skipped 000592 1786564526.603200 fallocate 0x018284 thru 0x01b977 (0x36f4 bytes) 000593 1786564526.730842 write 0x0265b1 thru 0x02b553 (0x4fa3 bytes) 000595 1786564526.744554 write 0x010d7e thru 0x01ba6e (0xacf1 bytes) 000597 1786564526.759750 read 0x0169c1 thru 0x019003 (0x2643 bytes) 000598 1786564526.765949 MIRROR_RESYNC 6 mirrors 000599 1786564528.996827 trunc from 0x02b554 to 0x02ae39 (0x71a bytes) 000600 1786564529.063288 trunc from 0x02ae3a to 0x023d20 (0x7119 bytes) 000601 1786564529.100016 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x23d21): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23d21): Mirror 5: 0xb6aa4700 (stale) Mirror 6: 0xb6aa4700 (stale) Mirror 7: 0xb6aa4700 (stale) Mirror 8: 0xb6aa4700 (stale) Mirror 9: 0xb6aa4700 (stale) CRC-32 checksum value for chunk [0, 0x23d21): Mirror 1: 0xb6aa4700 Mirror 10: 0xb6aa4700 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23d21: skipped 000602 1786564530.361024 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x23d21): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23d21): Mirror 5: 0xb6aa4700 (stale) Mirror 6: 0xb6aa4700 (stale) Mirror 7: 0xb6aa4700 (stale) Mirror 8: 0xb6aa4700 (stale) Mirror 9: 0xb6aa4700 (stale) CRC-32 checksum value for chunk [0, 0x23d21): Mirror 1: 0xb6aa4700 Mirror 10: 0xb6aa4700 Mirror 11: 0xb6aa4700 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23d21: skipped 000603 1786564531.871985 trunc from 0x015450 to 0x020144 (0xacf5 bytes) 000604 1786564531.922080 read 0x00aed6 thru 0x014898 (0x99c3 bytes) 000605 1786564531.930822 trunc from 0x023d21 to 0x01b144 (0x8bdc bytes) 000606 1786564531.957975 punch from 0x007555 to 0x00824e (0xcfa bytes) 000607 1786564531.964286 trunc from 0x01b145 to 0x02deda (0x12d96 bytes) 000608 1786564531.994727 mapwrite 0x02114a thru 0x024d76 (0x3c2d bytes) 000609 1786564532.049471 fallocate 0x012559 thru 0x0135ec (0x1094 bytes) 000610 1786564532.069321 trunc from 0x02acdb to 0x02deda (0x3200 bytes) 000611 1786564532.084558 trunc from 0x0283b2 to 0x02deda (0x5b29 bytes) 000612 1786564532.098969 punch from 0x0076c9 to 0x011408 (0x9d40 bytes) 000613 1786564532.118477 mapread 0x011418 thru 0x01edf5 (0xd9de bytes) 000614 1786564532.182172 mapread 0x00318c thru 0x00ca66 (0x98db bytes) 000615 1786564532.206609 write 0x000ad2 thru 0x00cad6 (0xc005 bytes) 000616 1786564532.225327 trunc from 0x022eb4 to 0x02deda (0xb027 bytes) 000617 1786564532.239761 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2dedb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dedb): Mirror 5: 0xf155e992 (stale) Mirror 6: 0xf155e992 (stale) Mirror 7: 0xf155e992 (stale) Mirror 8: 0xf155e992 (stale) Mirror 9: 0xf155e992 (stale) Mirror 10: 0xd43d7eb1 (stale) Mirror 11: 0xd43d7eb1 (stale) CRC-32 checksum value for chunk [0, 0x2dedb): Mirror 1: 0xce2e75f6 Mirror 12: 0xce2e75f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dedb: skipped 000618 1786564533.364320 mapread 0x01f1d5 thru 0x02deda (0xed06 bytes) 000619 1786564533.426271 read 0x02af18 thru 0x02deda (0x2fc3 bytes) 000620 1786564533.433537 punch from 0x004bca to 0x00e981 (0x9db8 bytes) 000621 1786564533.512160 write 0x00d033 thru 0x014213 (0x71e1 bytes) 000622 1786564533.556923 trunc from 0x02dedb to 0x00cb51 (0x21389 bytes) 000623 1786564533.616307 read 0x001eb0 thru 0x005806 (0x3957 bytes) 000624 1786564533.646608 mapwrite 0x00cbca thru 0x00fde8 (0x321f bytes) 000625 1786564533.693362 fallocate 0x0136ef thru 0x015f7d (0x288f bytes) 000626 1786564533.700460 mapwrite 0x016764 thru 0x01d38c (0x6c29 bytes) 000627 1786564533.769522 read 0x003ca4 thru 0x011bf5 (0xdf52 bytes) 000628 1786564533.770989 write 0x02a28f thru 0x03643d (0xc1af bytes) 000629 1786564533.775811 read 0x020c56 thru 0x024206 (0x35b1 bytes) 000631 1786564533.793130 mapread 0x00c242 thru 0x00e311 (0x20d0 bytes) 000632 1786564533.798269 read 0x0115aa thru 0x01b297 (0x9cee bytes) 000633 1786564533.799444 trunc from 0x026f7f to 0x031f0b (0xaf8d bytes) 000634 1786564533.817966 mapwrite 0x009c68 thru 0x00f871 (0x5c0a bytes) 000635 1786564533.832705 fallocate 0x01c5fc thru 0x020487 (0x3e8c bytes) 000636 1786564533.839579 trunc from 0x03643e to 0x008217 (0x2e226 bytes) 000637 1786564533.859202 fallocate 0x000fcb thru 0x007ef2 (0x6f28 bytes) 000639 1786564533.872985 mapwrite 0x037c89 thru 0x039f19 (0x2291 bytes) 000640 1786564533.921809 fallocate 0x028b40 thru 0x0350ea (0xc5ab bytes) 000641 1786564533.927678 punch from 0x039728 to 0x039f19 (0x7f2 bytes) 000642 1786564533.935890 read 0x03836b thru 0x039f19 (0x1baf bytes) 000643 1786564533.948996 trunc from 0x039f1a to 0x0017ee (0x3872b bytes) 000644 1786564533.986988 read 0x0007bc thru 0x0017ee (0x1033 bytes) 000645 1786564533.989357 trunc from 0x0017ef to 0x03961a (0x37e2c bytes) 000647 1786564534.024441 write 0x017c33 thru 0x021faf (0xa37d bytes) 000648 1786564534.047454 punch from 0x026756 to 0x029aa2 (0x334d bytes) 000649 1786564534.053073 trunc from 0x003220 to 0x00f164 (0xbf45 bytes) 000650 1786564534.066786 trunc from 0x02ed0f to 0x038a40 (0x9d32 bytes) 000651 1786564534.080084 trunc from 0x021ede to 0x0235f1 (0x1714 bytes) 000652 1786564534.124628 mapwrite 0x01ba5d thru 0x01c715 (0xcb9 bytes) 000653 1786564534.159568 fallocate 0x01ccdf thru 0x02b7b8 (0xeada bytes) 000654 1786564534.174632 write 0x0238ca thru 0x030d16 (0xd44d bytes) 000655 1786564534.196498 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3961b): Mirror 5: 0xcfe08f87 (stale) Mirror 7: 0xcfe08f87 (stale) Mirror 8: 0xcfe08f87 (stale) Mirror 9: 0xcfe08f87 (stale) Mirror 10: 0xc5df74a4 (stale) Mirror 11: 0xc5df74a4 (stale) Mirror 12: 0x5ef0d190 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3961b): Mirror 5: 0xcfe08f87 (stale) Mirror 7: 0xcfe08f87 (stale) Mirror 8: 0xcfe08f87 (stale) Mirror 9: 0xcfe08f87 (stale) Mirror 10: 0xc5df74a4 (stale) Mirror 11: 0xc5df74a4 (stale) Mirror 12: 0x5ef0d190 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3961b: skipped 000656 1786564534.844747 write 0x03d731 thru 0x03ffff (0x28cf bytes) 000657 1786564534.855576 mapwrite 0x014c00 thru 0x019c38 (0x5039 bytes) 000658 1786564534.927843 write 0x0201cb thru 0x02c81e (0xc654 bytes) 000659 1786564534.955785 trunc from 0x040000 to 0x03bf7c (0x4083 bytes) 000660 1786564534.987929 read 0x00395f thru 0x010477 (0xcb19 bytes) 000661 1786564535.000045 trunc from 0x03bf7d to 0x00db61 (0x2e41b bytes) 000662 1786564535.024855 punch from 0x00bdeb to 0x00db61 (0x1d77 bytes) 000663 1786564535.031069 mapread 0x0064ed thru 0x00d5f8 (0x710c bytes) 000664 1786564535.039318 trunc from 0x00db62 to 0x022a82 (0x14f21 bytes) 000666 1786564535.057901 trunc from 0x01289d to 0x01e4bd (0xbc21 bytes) 000667 1786564535.066890 mapwrite 0x03b993 thru 0x03cc6c (0x12da bytes) 000668 1786564535.123354 read 0x035732 thru 0x0366a8 (0xf77 bytes) 000669 1786564535.125047 fallocate 0x02eb32 thru 0x03c035 (0xd504 bytes) 000670 1786564535.131205 trunc from 0x01407f to 0x014d0f (0xc91 bytes) 000671 1786564535.136939 write 0x03525e thru 0x03ab96 (0x5939 bytes) 000672 1786564535.139149 write 0x0094a2 thru 0x01035b (0x6eba bytes) 000673 1786564535.149831 trunc from 0x03cc6d to 0x014c3d (0x2802f bytes) 000674 1786564535.171996 write 0x00531a thru 0x014d2a (0xfa11 bytes) 000675 1786564535.185506 read 0x014aec thru 0x014d2a (0x23f bytes) 000676 1786564535.186571 trunc from 0x014d2b to 0x02452b (0xf801 bytes) 000677 1786564535.209107 fallocate 0x00f065 thru 0x01c537 (0xd4d3 bytes) 000678 1786564535.215816 trunc from 0x02452c to 0x01946e (0xb0bd bytes) 000679 1786564535.235898 read 0x00207b thru 0x00daa8 (0xba2e bytes) 000681 1786564535.243658 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1946f): Mirror 5: 0x292cf5e (stale) Mirror 8: 0x292cf5e (stale) Mirror 9: 0x292cf5e (stale) Mirror 10: 0x292cf5e (stale) Mirror 11: 0x292cf5e (stale) Mirror 12: 0x8236a837 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1946f): Mirror 5: 0x292cf5e (stale) Mirror 8: 0x292cf5e (stale) Mirror 9: 0x292cf5e (stale) Mirror 10: 0x292cf5e (stale) Mirror 11: 0x292cf5e (stale) Mirror 12: 0x8236a837 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1946f: skipped 000682 1786564535.920573 mapread 0x006159 thru 0x0134ca (0xd372 bytes) 000683 1786564535.978343 mapread 0x010fe3 thru 0x01946e (0x848c bytes) 000684 1786564535.981485 trunc from 0x006ab5 to 0x00a584 (0x3ad0 bytes) 000685 1786564535.995561 fallocate 0x00d02b thru 0x017553 (0xa529 bytes) 000686 1786564536.004067 write 0x00ed5a thru 0x0185a5 (0x984c bytes) 000687 1786564536.018330 fallocate 0x01ef5b thru 0x02363a (0x46e0 bytes) 000688 1786564536.029089 read 0x00814c thru 0x010556 (0x840b bytes) 000689 1786564536.040324 mapread 0x008466 thru 0x009218 (0xdb3 bytes) 000690 1786564536.043047 fallocate 0x002cf3 thru 0x00a7af (0x7abd bytes) 000691 1786564536.051827 mapwrite 0x032cad thru 0x0383df (0x5733 bytes) 000693 1786564536.205197 write 0x030c4f thru 0x0393ad (0x875f bytes) 000694 1786564536.209285 mapwrite 0x00afd5 thru 0x012f7a (0x7fa6 bytes) 000695 1786564536.263564 read 0x025177 thru 0x02accf (0x5b59 bytes) 000696 1786564536.296871 fallocate 0x00641e thru 0x0124e8 (0xc0cb bytes) 000697 1786564536.308306 read 0x01b84e thru 0x02248b (0x6c3e bytes) 000698 1786564536.309641 mapwrite 0x01db7c thru 0x028000 (0xa485 bytes) 000699 1786564536.341044 fallocate 0x0197d8 thru 0x01bcee (0x2517 bytes) 000700 1786564536.351808 mapwrite 0x03f55f thru 0x03ffff (0xaa1 bytes) 000701 1786564536.398575 mapwrite 0x030651 thru 0x03f2e9 (0xec99 bytes) 000702 1786564536.427737 MIRROR_RESYNC 7 mirrors 000703 1786564539.045852 fallocate 0x029194 thru 0x032ea9 (0x9d16 bytes) 000704 1786564539.101552 write 0x02df9d thru 0x02f4bb (0x151f bytes) 000705 1786564539.122133 write 0x02ef21 thru 0x03aa7e (0xbb5e bytes) 000706 1786564539.132901 fallocate 0x037f8c thru 0x03ffff (0x8074 bytes) 000707 1786564539.142727 mapread 0x001163 thru 0x00c374 (0xb212 bytes) 000708 1786564539.166778 trunc from 0x040000 to 0x035820 (0xa7df bytes) 000710 1786564539.201821 mapwrite 0x005a9f thru 0x00fba0 (0xa102 bytes) 000711 1786564539.265349 write 0x0275e1 thru 0x02d3d7 (0x5df7 bytes) 000712 1786564539.273477 mapread 0x02c958 thru 0x034435 (0x7ade bytes) 000714 1786564539.277762 write 0x01d156 thru 0x021af2 (0x499d bytes) 000715 1786564539.311799 mapread 0x0303d2 thru 0x032a45 (0x2674 bytes) 000717 1786564539.315516 trunc from 0x035821 to 0x026553 (0xf2cd bytes) 000718 1786564539.372528 mapwrite 0x03d076 thru 0x03d292 (0x21d bytes) 000719 1786564539.496388 trunc from 0x03d293 to 0x016b73 (0x2671f bytes) 000720 1786564539.519650 mapread 0x008793 thru 0x00d472 (0x4ce0 bytes) 000721 1786564539.522601 trunc from 0x013f76 to 0x016b73 (0x2bfe bytes) 000722 1786564539.531826 trunc from 0x00bb1e to 0x00f449 (0x392c bytes) 000723 1786564539.549272 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16b74): Mirror 8: 0xc7440222 (stale) Mirror 9: 0xc7440222 (stale) Mirror 10: 0xc7440222 (stale) Mirror 11: 0xc7440222 (stale) Mirror 12: 0xc7440222 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16b74): Mirror 8: 0xc7440222 (stale) Mirror 9: 0xc7440222 (stale) Mirror 10: 0xc7440222 (stale) Mirror 11: 0xc7440222 (stale) Mirror 12: 0xc7440222 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16b74: skipped 000724 1786564540.155936 trunc from 0x016b74 to 0x030b9f (0x1a02c bytes) 000725 1786564540.197323 punch from 0x010901 to 0x019c32 (0x9332 bytes) 000726 1786564540.220569 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30ba0): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ba0): Mirror 8: 0xc6321505 (stale) Mirror 9: 0xc6321505 (stale) Mirror 10: 0xc6321505 (stale) Mirror 11: 0xc6321505 (stale) Mirror 12: 0xc6321505 (stale) CRC-32 checksum value for chunk [0, 0x30ba0): Mirror 1: 0x5571c27 Mirror 13: 0x5571c27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ba0: skipped 000727 1786564541.547220 trunc from 0x021ced to 0x0230e5 (0x13f9 bytes) 000728 1786564541.642022 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30ba0): Mirror 9: 0xc6321505 (stale) Mirror 10: 0xc6321505 (stale) Mirror 11: 0xc6321505 (stale) Mirror 12: 0xc6321505 (stale) Mirror 13: 0x5571c27 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ba0): Mirror 9: 0xc6321505 (stale) Mirror 10: 0xc6321505 (stale) Mirror 11: 0xc6321505 (stale) Mirror 12: 0xc6321505 (stale) Mirror 13: 0x5571c27 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ba0: skipped 000729 1786564542.327984 mapread 0x0240e8 thru 0x029a2c (0x5945 bytes) 000730 1786564542.389970 trunc from 0x0142c9 to 0x01d6ec (0x9424 bytes) 000731 1786564542.396842 mapread 0x0246f3 thru 0x025236 (0xb44 bytes) 000732 1786564542.398120 write 0x0017cd thru 0x0028f5 (0x1129 bytes) 000734 1786564542.427476 mapread 0x005a2e thru 0x00c6dc (0x6caf bytes) 000736 1786564542.455594 read 0x02884c thru 0x030b9f (0x8354 bytes) 000737 1786564542.473940 mapread 0x00d6b7 thru 0x014d48 (0x7692 bytes) 000739 1786564542.488225 fallocate 0x01a5ec thru 0x01c8b8 (0x22cd bytes) 000740 1786564542.497095 read 0x0275f3 thru 0x02cb09 (0x5517 bytes) 000741 1786564542.498972 write 0x02f29d thru 0x03b3aa (0xc10e bytes) 000742 1786564542.506227 read 0x00812a thru 0x011f2c (0x9e03 bytes) 000743 1786564542.511629 trunc from 0x012ea3 to 0x01a17b (0x72d9 bytes) 000744 1786564542.526602 mapwrite 0x017449 thru 0x0246a4 (0xd25c bytes) 000745 1786564542.597819 punch from 0x02e0c0 to 0x02ff1a (0x1e5b bytes) 000746 1786564542.613137 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b3ab): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b3ab): Mirror 9: 0xf9d7d131 (stale) Mirror 10: 0xf9d7d131 (stale) Mirror 11: 0xf9d7d131 (stale) Mirror 12: 0xf9d7d131 (stale) Mirror 13: 0x77b691bd (stale) CRC-32 checksum value for chunk [0, 0x3b3ab): Mirror 1: 0xbdfb5d22 Mirror 14: 0xbdfb5d22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b3ab: skipped 000747 1786564544.282427 mapwrite 0x01c447 thru 0x020861 (0x441b bytes) 000748 1786564544.552803 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b3ab): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b3ab): Mirror 9: 0xf9d7d131 (stale) Mirror 10: 0xf9d7d131 (stale) Mirror 11: 0xf9d7d131 (stale) Mirror 12: 0xf9d7d131 (stale) Mirror 13: 0x77b691bd (stale) Mirror 14: 0xbdfb5d22 (stale) CRC-32 checksum value for chunk [0, 0x3b3ab): Mirror 1: 0x5ffff35b Mirror 15: 0x5ffff35b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b3ab: skipped 000749 1786564546.168057 trunc from 0x030660 to 0x035d76 (0x5717 bytes) 000751 1786564546.234280 mapread 0x031529 thru 0x032903 (0x13db bytes) 000752 1786564546.248171 fallocate 0x01554f thru 0x01874e (0x3200 bytes) 000753 1786564546.257214 trunc from 0x03b3ab to 0x008c43 (0x32767 bytes) 000754 1786564546.293120 read 0x00803e thru 0x008c43 (0xc06 bytes) 000755 1786564546.300121 trunc from 0x008c44 to 0x02ccf8 (0x240b5 bytes) 000758 1786564546.333259 punch from 0x02a017 to 0x02ccf8 (0x2ce2 bytes) 000759 1786564546.344725 fallocate 0x004f57 thru 0x012146 (0xd1f0 bytes) 000761 1786564546.361087 write 0x01c998 thru 0x028446 (0xbaaf bytes) 000762 1786564546.381309 write 0x03e5f7 thru 0x03ffff (0x1a09 bytes) 000763 1786564546.383956 write 0x030f8c thru 0x033af5 (0x2b6a bytes) 000764 1786564546.395825 trunc from 0x040000 to 0x037744 (0x88bb bytes) 000765 1786564546.415295 read 0x0338e6 thru 0x037744 (0x3e5f bytes) 000766 1786564546.429970 mapread 0x0309fe thru 0x037744 (0x6d47 bytes) 000769 1786564546.434255 trunc from 0x01d530 to 0x022684 (0x5155 bytes) 000770 1786564546.473246 trunc from 0x008851 to 0x015395 (0xcb45 bytes) 000771 1786564546.486314 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37745): Mirror 9: 0x8f9658c (stale) Mirror 10: 0x8f9658c (stale) Mirror 12: 0x8f9658c (stale) Mirror 13: 0x8e181caa (stale) Mirror 14: 0x9feae8f2 (stale) Mirror 15: 0x3b0ee3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37745): Mirror 9: 0x8f9658c (stale) Mirror 10: 0x8f9658c (stale) Mirror 12: 0x8f9658c (stale) Mirror 13: 0x8e181caa (stale) Mirror 14: 0x9feae8f2 (stale) Mirror 15: 0x3b0ee3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37745: skipped 000772 1786564547.092571 mapwrite 0x001690 thru 0x00eb67 (0xd4d8 bytes) 000773 1786564547.135650 fallocate 0x011c91 thru 0x0131a4 (0x1514 bytes) 000774 1786564547.141017 trunc from 0x037745 to 0x034df6 (0x294e bytes) 000775 1786564547.169827 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x34df7): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34df7): Mirror 9: 0x93ab1b88 (stale) Mirror 10: 0x93ab1b88 (stale) Mirror 12: 0x93ab1b88 (stale) Mirror 13: 0x4fca3645 (stale) Mirror 14: 0xa6b89009 (stale) Mirror 15: 0x5391cc5b (stale) CRC-32 checksum value for chunk [0, 0x34df7): Mirror 1: 0x6603600b Mirror 16: 0x6603600b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34df7: skipped 000776 1786564548.451111 mapwrite 0x020381 thru 0x021db0 (0x1a30 bytes) 000777 1786564548.563016 mapread 0x01c203 thru 0x01f908 (0x3706 bytes) 000778 1786564548.566540 fallocate 0x01de43 thru 0x025183 (0x7341 bytes) 000779 1786564548.574123 mapwrite 0x03b264 thru 0x03ffff (0x4d9c bytes) 000780 1786564548.668458 read 0x027bb3 thru 0x028d48 (0x1196 bytes) 000781 1786564548.669296 trunc from 0x0066c1 to 0x01089f (0xa1df bytes) 000782 1786564548.686158 read 0x006285 thru 0x00dd87 (0x7b03 bytes) 000783 1786564548.711819 mapread 0x0027d8 thru 0x00ef3a (0xc763 bytes) 000784 1786564548.715781 fallocate 0x012f89 thru 0x0171b6 (0x422e bytes) 000785 1786564548.726109 trunc from 0x040000 to 0x01f4e3 (0x20b1c bytes) 000786 1786564548.749276 trunc from 0x01f4e4 to 0x0012a9 (0x1e23a bytes) 000787 1786564548.770087 fallocate 0x021a86 thru 0x027ac7 (0x6042 bytes) 000790 1786564548.777616 read 0x00ea7a thru 0x01ae16 (0xc39d bytes) 000791 1786564548.792611 punch from 0x023017 to 0x027ac7 (0x4ab1 bytes) 000792 1786564548.802731 MIRROR_RESYNC 8 mirrors 000793 1786564550.500040 trunc from 0x027ac8 to 0x01a138 (0xd98f bytes) 000794 1786564550.553987 mapread 0x011e96 thru 0x016dd9 (0x4f44 bytes) 000795 1786564550.578239 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a139): Mirror 9: 0x6061b5b3 (stale) Mirror 12: 0x6061b5b3 (stale) Mirror 13: 0x6061b5b3 (stale) Mirror 14: 0x6061b5b3 (stale) Mirror 15: 0x6061b5b3 (stale) Mirror 16: 0x6061b5b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a139): Mirror 9: 0x6061b5b3 (stale) Mirror 12: 0x6061b5b3 (stale) Mirror 13: 0x6061b5b3 (stale) Mirror 14: 0x6061b5b3 (stale) Mirror 15: 0x6061b5b3 (stale) Mirror 16: 0x6061b5b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a139: skipped 000796 1786564551.093855 write 0x010beb thru 0x01614e (0x5564 bytes) 000797 1786564551.123919 fallocate 0x011ec3 thru 0x01ab13 (0x8c51 bytes) 000800 1786564551.136056 write 0x01776c thru 0x019904 (0x2199 bytes) 000801 1786564551.172381 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a139): Mirror 12: 0x6061b5b3 (stale) Mirror 13: 0x6061b5b3 (stale) Mirror 14: 0x6061b5b3 (stale) Mirror 15: 0x6061b5b3 (stale) Mirror 16: 0x6061b5b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a139): Mirror 12: 0x6061b5b3 (stale) Mirror 13: 0x6061b5b3 (stale) Mirror 14: 0x6061b5b3 (stale) Mirror 15: 0x6061b5b3 (stale) Mirror 16: 0x6061b5b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a139: skipped 000802 1786564551.727795 fallocate 0x036243 thru 0x03dbcc (0x798a bytes) 000806 1786564551.738571 trunc from 0x01a139 to 0x03830f (0x1e1d7 bytes) 000807 1786564551.766526 punch from 0x023dbb to 0x02e071 (0xa2b7 bytes) 000808 1786564551.773693 fallocate 0x00020f thru 0x010032 (0xfe24 bytes) 000809 1786564551.781328 mapread 0x01be6f thru 0x02a2ee (0xe480 bytes) 000810 1786564551.810769 mapwrite 0x03e131 thru 0x03ffff (0x1ecf bytes) 000811 1786564551.875880 MIRROR_RESYNC 6 mirrors 000812 1786564553.933497 mapwrite 0x00f914 thru 0x017cae (0x839b bytes) 000813 1786564554.083928 mapwrite 0x00cd9a thru 0x011cd6 (0x4f3d bytes) 000814 1786564554.110845 trunc from 0x040000 to 0x00055a (0x3faa5 bytes) 000815 1786564554.186147 trunc from 0x00055b to 0x021afc (0x215a2 bytes) 000816 1786564554.211349 read 0x00f8bb thru 0x01d81d (0xdf63 bytes) 000817 1786564554.239687 trunc from 0x004e0a to 0x00e033 (0x922a bytes) 000818 1786564554.253769 punch from 0x019770 to 0x021afc (0x838d bytes) 000819 1786564554.266916 mapwrite 0x01f481 thru 0x02178b (0x230b bytes) 000820 1786564554.326666 trunc from 0x021afd to 0x00e0e7 (0x13a15 bytes) 000821 1786564554.376659 MIRROR_RESYNC 6 mirrors 000822 1786564555.955482 mapread 0x007c39 thru 0x0089b4 (0xd7c bytes) 000823 1786564555.973152 trunc from 0x009d29 to 0x00e0e7 (0x43bf bytes) 000824 1786564556.009126 mapwrite 0x00d9d2 thru 0x01cb90 (0xf1bf bytes) 000825 1786564556.169858 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1cb91): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cb91): Mirror 12: 0x47705f87 (stale) Mirror 13: 0x47705f87 (stale) Mirror 14: 0x47705f87 (stale) Mirror 15: 0x47705f87 (stale) Mirror 16: 0x47705f87 (stale) CRC-32 checksum value for chunk [0, 0x1cb91): Mirror 1: 0x6b153711 Mirror 17: 0x6b153711 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cb91: skipped 000826 1786564557.239462 read 0x00b307 thru 0x018e66 (0xdb60 bytes) 000827 1786564557.254929 write 0x03a1bb thru 0x03affe (0xe44 bytes) 000828 1786564557.332240 write 0x013678 thru 0x01d210 (0x9b99 bytes) 000829 1786564557.389060 MIRROR_RESYNC 7 mirrors 000830 1786564559.417049 write 0x03d9a9 thru 0x03ffff (0x2657 bytes) 000831 1786564559.485435 read 0x039df1 thru 0x039f44 (0x154 bytes) 000832 1786564559.503458 write 0x032bc1 thru 0x03aaa9 (0x7ee9 bytes) 000834 1786564559.518545 mapwrite 0x00752a thru 0x00c14a (0x4c21 bytes) 000835 1786564559.576178 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf0558a03 (stale) Mirror 13: 0xf0558a03 (stale) Mirror 14: 0xf0558a03 (stale) Mirror 15: 0xf0558a03 (stale) Mirror 16: 0xf0558a03 (stale) Mirror 17: 0xf0558a03 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x26f1bef6 Mirror 18: 0x26f1bef6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000836 1786564561.286879 write 0x0342ef thru 0x03e9b6 (0xa6c8 bytes) 000838 1786564561.394770 trunc from 0x040000 to 0x023d33 (0x1c2cc bytes) 000839 1786564561.425364 mapread 0x01a2c1 thru 0x023d33 (0x9a73 bytes) 000840 1786564561.458020 trunc from 0x023d34 to 0x0177e0 (0xc553 bytes) 000842 1786564561.481483 trunc from 0x0177e1 to 0x02f8b8 (0x180d8 bytes) 000843 1786564561.500749 mapread 0x02121b thru 0x02c361 (0xb147 bytes) 000844 1786564561.526259 fallocate 0x039bcd thru 0x03ffff (0x6433 bytes) 000845 1786564561.535996 mapread 0x00e502 thru 0x0155b3 (0x70b2 bytes) 000846 1786564561.547447 write 0x0066c0 thru 0x00f4cc (0x8e0d bytes) 000847 1786564561.558695 read 0x02548d thru 0x02a85f (0x53d3 bytes) 000848 1786564561.560025 mapwrite 0x00eb0a thru 0x016b86 (0x807d bytes) 000849 1786564561.596347 MIRROR_RESYNC 8 mirrors 000850 1786564563.742863 fallocate 0x03454f thru 0x03e5d6 (0xa088 bytes) 000851 1786564563.785713 fallocate 0x0296f8 thru 0x02e921 (0x522a bytes) 000854 1786564563.800870 mapwrite 0x039eb6 thru 0x03ffff (0x614a bytes) 000855 1786564563.911711 trunc from 0x002d35 to 0x0054cf (0x279b bytes) 000856 1786564563.921357 write 0x02a7d2 thru 0x02efb4 (0x47e3 bytes) 000857 1786564563.943051 MIRROR_RESYNC 8 mirrors 000858 1786564566.275317 write 0x01ccf7 thru 0x01d97c (0xc86 bytes) 000859 1786564566.338695 trunc from 0x040000 to 0x0040f8 (0x3bf07 bytes) 000861 1786564566.390329 mapwrite 0x002602 thru 0x008bc6 (0x65c5 bytes) 000862 1786564566.530609 write 0x03fa43 thru 0x03ffff (0x5bd bytes) 000863 1786564566.532665 fallocate 0x001a15 thru 0x00b964 (0x9f50 bytes) 000864 1786564566.541347 fallocate 0x000451 thru 0x0094ff (0x90af bytes) 000865 1786564566.550330 mapread 0x0156f3 thru 0x022e7f (0xd78d bytes) 000867 1786564566.576332 mapwrite 0x021b81 thru 0x02c378 (0xa7f8 bytes) 000868 1786564566.628124 punch from 0x008b17 to 0x015914 (0xcdfe bytes) 000869 1786564566.636279 mapread 0x03289f thru 0x03a76e (0x7ed0 bytes) 000870 1786564566.650240 punch from 0x01f9a0 to 0x02209b (0x26fc bytes) 000871 1786564566.657761 read 0x008896 thru 0x00e99b (0x6106 bytes) 000872 1786564566.672214 write 0x026fca thru 0x0346cc (0xd703 bytes) 000874 1786564566.681973 mapwrite 0x02868a thru 0x0319eb (0x9362 bytes) 000875 1786564566.708917 mapread 0x006ea8 thru 0x007fe2 (0x113b bytes) 000876 1786564566.712046 mapread 0x025a9f thru 0x02668c (0xbee bytes) 000877 1786564566.717757 trunc from 0x0213a8 to 0x02b0dd (0x9d36 bytes) 000878 1786564566.727630 trunc from 0x002fbd to 0x00cef9 (0x9f3d bytes) 000879 1786564566.736415 read 0x002ff9 thru 0x009886 (0x688e bytes) 000880 1786564566.767530 mapwrite 0x012976 thru 0x0170a2 (0x472d bytes) 000881 1786564566.790659 trunc from 0x00381b to 0x0128d7 (0xf0bd bytes) 000883 1786564566.798156 trunc from 0x040000 to 0x004cea (0x3b315 bytes) 000884 1786564566.815626 trunc from 0x004ceb to 0x0386ce (0x339e4 bytes) 000885 1786564566.843072 mapwrite 0x03381a thru 0x038f2b (0x5712 bytes) 000886 1786564566.950385 read 0x00f267 thru 0x018c0f (0x99a9 bytes) 000888 1786564566.985423 fallocate 0x022c8b thru 0x025f47 (0x32bd bytes) 000889 1786564566.998237 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f2c): Mirror 13: 0x6515049e (stale) Mirror 14: 0x6515049e (stale) Mirror 15: 0x6515049e (stale) Mirror 16: 0x6515049e (stale) Mirror 17: 0x6515049e (stale) Mirror 18: 0x6515049e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f2c): Mirror 13: 0x6515049e (stale) Mirror 14: 0x6515049e (stale) Mirror 15: 0x6515049e (stale) Mirror 16: 0x6515049e (stale) Mirror 17: 0x6515049e (stale) Mirror 18: 0x6515049e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f2c: skipped 000890 1786564567.514168 trunc from 0x038f2c to 0x036156 (0x2dd5 bytes) 000891 1786564567.544892 mapread 0x026322 thru 0x02cb6d (0x684c bytes) 000892 1786564567.568198 mapwrite 0x033981 thru 0x03dc48 (0xa2c8 bytes) 000893 1786564567.629475 write 0x006178 thru 0x00bdc6 (0x5c4f bytes) 000894 1786564567.647116 read 0x034d23 thru 0x03d9e7 (0x8cc5 bytes) 000895 1786564567.648137 mapwrite 0x0256b2 thru 0x026d0a (0x1659 bytes) 000896 1786564567.669901 mapread 0x0236d1 thru 0x02d53b (0x9e6b bytes) 000898 1786564567.673355 fallocate 0x001bac thru 0x00b300 (0x9755 bytes) 000899 1786564567.683968 read 0x01c92a thru 0x026df6 (0xa4cd bytes) 000900 1786564567.722954 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc49): Mirror 14: 0xcb78b936 (stale) Mirror 15: 0xcb78b936 (stale) Mirror 16: 0xcb78b936 (stale) Mirror 17: 0xcb78b936 (stale) Mirror 18: 0xcb78b936 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc49): Mirror 14: 0xcb78b936 (stale) Mirror 15: 0xcb78b936 (stale) Mirror 16: 0xcb78b936 (stale) Mirror 17: 0xcb78b936 (stale) Mirror 18: 0xcb78b936 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc49: skipped 000901 1786564568.252834 write 0x03e196 thru 0x03ffff (0x1e6a bytes) 000903 1786564568.261518 trunc from 0x018a64 to 0x02755f (0xeafc bytes) 000904 1786564568.266608 read 0x01eaee thru 0x02b40f (0xc922 bytes) 000906 1786564568.283318 fallocate 0x01b388 thru 0x0283f7 (0xd070 bytes) 000907 1786564568.295493 trunc from 0x040000 to 0x01d242 (0x22dbd bytes) 000908 1786564568.337086 mapwrite 0x01845e thru 0x019c39 (0x17dc bytes) 000909 1786564568.401513 mapread 0x0146cc thru 0x0177bb (0x30f0 bytes) 000910 1786564568.418367 read 0x0088c2 thru 0x00edb0 (0x64ef bytes) 000911 1786564568.433212 trunc from 0x01d243 to 0x02779f (0xa55d bytes) 000912 1786564568.462137 trunc from 0x0277a0 to 0x03a6a5 (0x12f06 bytes) 000913 1786564568.505545 read 0x02695b thru 0x0336ce (0xcd74 bytes) 000914 1786564568.520960 mapread 0x02c03e thru 0x037caa (0xbc6d bytes) 000915 1786564568.523272 mapwrite 0x00373e thru 0x00931d (0x5be0 bytes) 000916 1786564568.549866 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a6a6): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a6a6): Mirror 14: 0xdedff7b9 (stale) Mirror 15: 0xdedff7b9 (stale) Mirror 16: 0xdedff7b9 (stale) Mirror 17: 0xdedff7b9 (stale) Mirror 18: 0xdedff7b9 (stale) CRC-32 checksum value for chunk [0, 0x3a6a6): Mirror 1: 0x2cf84576 Mirror 19: 0x2cf84576 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a6a6: skipped 000917 1786564569.757724 read 0x018224 thru 0x0221ff (0x9fdc bytes) 000918 1786564569.768052 fallocate 0x00e9bc thru 0x0120c2 (0x3707 bytes) 000919 1786564569.814672 write 0x02e9c2 thru 0x036c12 (0x8251 bytes) 000921 1786564569.836287 write 0x01aa72 thru 0x025589 (0xab18 bytes) 000922 1786564569.850044 mapwrite 0x02447b thru 0x025be1 (0x1767 bytes) 000923 1786564569.868372 mapwrite 0x035e59 thru 0x03ffff (0xa1a7 bytes) 000924 1786564569.934229 write 0x017981 thru 0x023a2f (0xc0af bytes) 000925 1786564569.957504 mapwrite 0x02a839 thru 0x02d856 (0x301e bytes) 000926 1786564570.021917 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xaa9af8d9 (stale) Mirror 16: 0xaa9af8d9 (stale) Mirror 17: 0xaa9af8d9 (stale) Mirror 18: 0xaa9af8d9 (stale) Mirror 19: 0xda34108c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xaa9af8d9 (stale) Mirror 16: 0xaa9af8d9 (stale) Mirror 17: 0xaa9af8d9 (stale) Mirror 18: 0xaa9af8d9 (stale) Mirror 19: 0xda34108c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000927 1786564570.729647 fallocate 0x005a0d thru 0x00fb72 (0xa166 bytes) 000928 1786564570.741209 write 0x00272b thru 0x004259 (0x1b2f bytes) 000929 1786564570.767152 read 0x03d92b thru 0x03ffff (0x26d5 bytes) 000930 1786564570.781030 fallocate 0x00539a thru 0x00ae41 (0x5aa8 bytes) 000931 1786564570.787296 fallocate 0x03eea6 thru 0x03ffff (0x115a bytes) 000932 1786564570.811954 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xaa9af8d9 (stale) Mirror 16: 0xaa9af8d9 (stale) Mirror 17: 0xaa9af8d9 (stale) Mirror 18: 0xaa9af8d9 (stale) Mirror 19: 0xda34108c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf72ebbf2 Mirror 20: 0xf72ebbf2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000933 1786564572.218263 punch from 0x03e8bf to 0x03ffff (0x1741 bytes) 000934 1786564572.298970 mapread 0x011ecd thru 0x01fe02 (0xdf36 bytes) 000935 1786564572.343834 mapread 0x00c7cb thru 0x012f05 (0x673b bytes) 000936 1786564572.349831 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xaa9af8d9 (stale) Mirror 16: 0xaa9af8d9 (stale) Mirror 17: 0xaa9af8d9 (stale) Mirror 18: 0xaa9af8d9 (stale) Mirror 19: 0xda34108c (stale) Mirror 20: 0xf72ebbf2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe3ef06c4 Mirror 21: 0xe3ef06c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000937 1786564573.825195 trunc from 0x040000 to 0x01be4f (0x241b0 bytes) 000939 1786564573.950231 trunc from 0x015fea to 0x01b625 (0x563c bytes) 000940 1786564573.968124 write 0x03f48a thru 0x03f9ab (0x522 bytes) 000941 1786564573.971221 trunc from 0x01b1ed to 0x022792 (0x75a6 bytes) 000942 1786564573.978836 read 0x020bbf thru 0x02bfcd (0xb40f bytes) 000943 1786564573.993494 read 0x0312fa thru 0x038a1f (0x7726 bytes) 000944 1786564574.028387 write 0x01516a thru 0x01e32f (0x91c6 bytes) 000945 1786564574.036010 trunc from 0x03f9ac to 0x0034ce (0x3c4dd bytes) 000947 1786564574.075990 write 0x02ab97 thru 0x032d47 (0x81b1 bytes) 000948 1786564574.081439 punch from 0x0233cd to 0x030f4a (0xdb7e bytes) 000949 1786564574.102949 trunc from 0x032d48 to 0x024e6f (0xded8 bytes) 000950 1786564574.120523 read 0x01ba6e thru 0x01c8c5 (0xe58 bytes) 000951 1786564574.137963 trunc from 0x020dce to 0x024e6f (0x40a2 bytes) 000953 1786564574.145055 mapwrite 0x03d39c thru 0x03ffff (0x2c64 bytes) 000954 1786564574.231837 read 0x001279 thru 0x004cba (0x3a42 bytes) 000955 1786564574.235041 read 0x03e8ba thru 0x03ef37 (0x67e bytes) 000956 1786564574.240085 read 0x02d753 thru 0x03570a (0x7fb8 bytes) 000957 1786564574.250736 MIRROR_RESYNC 8 mirrors 000960 1786564576.250548 punch from 0x00d627 to 0x01b65f (0xe039 bytes) 000961 1786564576.296708 mapread 0x000cf1 thru 0x00aa16 (0x9d26 bytes) 000962 1786564576.338239 mapwrite 0x01a43b thru 0x0288f6 (0xe4bc bytes) 000964 1786564576.389170 write 0x00d8da thru 0x00e61e (0xd45 bytes) 000965 1786564576.391913 punch from 0x01f3b6 to 0x021338 (0x1f83 bytes) 000966 1786564576.398500 trunc from 0x0087e8 to 0x00e480 (0x5c99 bytes) 000967 1786564576.412199 write 0x007ffa thru 0x0091db (0x11e2 bytes) 000968 1786564576.418528 punch from 0x03a51b to 0x03ffff (0x5ae5 bytes) 000970 1786564576.425377 read 0x024a58 thru 0x027d31 (0x32da bytes) 000971 1786564576.426368 fallocate 0x02af79 thru 0x0300e3 (0x516b bytes) 000972 1786564576.432759 trunc from 0x040000 to 0x02513a (0x1aec5 bytes) 000973 1786564576.459176 fallocate 0x010002 thru 0x0182d9 (0x82d8 bytes) 000974 1786564576.465962 mapread 0x0090ca thru 0x017c56 (0xeb8d bytes) 000976 1786564576.481127 mapread 0x01da3c thru 0x02513a (0x76ff bytes) 000977 1786564576.489658 punch from 0x0125eb to 0x019fba (0x79d0 bytes) 000979 1786564576.497238 mapwrite 0x0113a9 thru 0x01ffe4 (0xec3c bytes) 000980 1786564576.538875 punch from 0x003c50 to 0x0078ef (0x3ca0 bytes) 000981 1786564576.549948 write 0x0094cf thru 0x00e98c (0x54be bytes) 000982 1786564576.556357 fallocate 0x0029bb thru 0x005cc8 (0x330e bytes) 000983 1786564576.573123 punch from 0x01c2a2 to 0x01d36b (0x10ca bytes) 000984 1786564576.585743 read 0x006db1 thru 0x010d9a (0x9fea bytes) 000985 1786564576.610340 trunc from 0x02513b to 0x02023c (0x4efe bytes) 000988 1786564576.641056 fallocate 0x006033 thru 0x013fe6 (0xdfb4 bytes) 000989 1786564576.659770 read 0x00d195 thru 0x0166ab (0x9517 bytes) 000990 1786564576.660733 mapread 0x0132b3 thru 0x01d7e2 (0xa530 bytes) 000991 1786564576.664280 mapread 0x0195d3 thru 0x02023c (0x6c6a bytes) 000993 1786564576.668555 write 0x02cf3d thru 0x0314e0 (0x45a4 bytes) 000994 1786564576.675038 fallocate 0x011dc1 thru 0x016a0c (0x4c4c bytes) 000995 1786564576.687644 mapwrite 0x00781c thru 0x015dca (0xe5af bytes) 000996 1786564576.751326 read 0x00bd02 thru 0x00c842 (0xb41 bytes) 000997 1786564576.752073 write 0x005ebd thru 0x009a60 (0x3ba4 bytes) 000998 1786564576.756692 mapread 0x02492e thru 0x02e08c (0x975f bytes) 000999 1786564576.770142 mapwrite 0x00e9b6 thru 0x012515 (0x3b60 bytes) 001000 1786564576.821915 read 0x004f2c thru 0x00c896 (0x796b bytes) All operations completed A-OK! PASS 70a (120s) == sanity-flr test 71: check mirror extend parent fid ==== 15:56:22 (1786564582) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x55:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 15:56:34 (1786564594) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246726 s, 8.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12341 s, 8.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 15:56:43 (1786564603) Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4288352 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8382816 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4494208 .. failed Extending file size to 8513376 .. Extending file size to 9214816 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8813152 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 61278 61280 61282 61283 61286 192.168.204.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg458-client.virtnet /mnt/lustre2 (opts:) 192.168.204.158@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg458-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:58:09 (1786564689) Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6378432 .. Extending file size to 4672800 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6509056 .. Extending file size to 7967680 .. Extending file size to 7307360 .. Extending file size to 7968576 .. Extending file size to 8144768 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8311264 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8154016 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8281184 .. Extending file size to 8419072 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8432672 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8544832 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8632256 .. Extending file size to 8997472 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62816 62818 62819 62820 62821 failed 192.168.204.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg458-client.virtnet /mnt/lustre2 (opts:) 192.168.204.158@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg458-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (92s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:59:41 (1786564781) Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2192 1269112 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25216 3581804 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17620 3589400 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42836 7171204 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2192 1269112 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25216 3581804 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17620 3589400 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42836 7171204 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167767 s, 12.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.247592 s, 8.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd3a9637c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg458-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:00:03 (1786564803) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0889172 s, 23.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 16:00:10 (1786564810) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:00:21 (1786564821) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134907 s, 15.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1f7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:00:32 (1786564832) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189603 s, 11.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:00:40 (1786564840) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129122 s, 16.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204c (40s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:01:20 (1786564880) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg458-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078366 s, 13.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:02:13 (1786564933) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg458-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0973717 s, 21.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:03:04 (1786564984) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg458-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg458-server: Pool lustre.test_204f created Adding targets to pool oleg458-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg458-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129442 s, 16.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg458-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg458-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg458-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:04:21 (1786565061) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:04:30 (1786565070) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:04:39 (1786565079) oleg458-server: Pool lustre.test_206 created Adding targets to pool oleg458-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg458-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg458-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg458-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg458-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:05:09 (1786565109) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131486 s, 8.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813835 s, 12.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x363f02be Mirror 3: 0x363f02be PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:05:21 (1786565121) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:05:25 (1786565125) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:05:29 (1786565129) Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0831395 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xae0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1190000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1190000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1190000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23a0000 Read this (4 bytes): BBBB 192.168.204.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg458-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:06:11 (1786565171) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0267134 s, 307 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:06:20 (1786565180) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845904 s, 12.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x87 lmm_fid: [0x200000405:0x87:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x87 lmm_fid: [0x200000405:0x87:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x122:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611271 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-12 16:06:22.000000000 -0400 Modify: 2026-08-12 16:06:22.000000000 -0400 Change: 2026-08-12 16:06:23.000000000 -0400 Birth: 2026-08-12 16:06:21.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:06:29 (1786565189) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340232 s, 30.8 MB/s oleg458-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg458-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg458-server: O/280000401/d31/13f: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:06:55 (1786565215) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.296567 s, 138 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0312039 s, 131 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 12 16:06 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 12 16:06 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:07:04 (1786565224) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:07:13 (1786565233) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1957 sec =========== 16:07:23 (1786565243) === sanity-flr: start cleanup 16:07:25 (1786565245) === === sanity-flr: finish cleanup 16:07:34 (1786565254) ===