-----============= acceptance-small: sanity-flr ============----- Thu Aug 13 09:30:51 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg317-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:31:10 (1786627870) === oleg317-client.virtnet: executing check_config_client /mnt/lustre oleg317-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg317-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa049c6b73000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa049c6b73000.idle_timeout=debug disable quota as required oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:31:56 (1786627916) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8246] == sanity-flr test 0a: lfs mirror create with -N option == 09:31:59 (1786627919) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:32:17 (1786627937) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:32:20 (1786627940) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:32:24 (1786627944) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:32:48 (1786627968) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:32:52 (1786627972) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 09:32:55 (1786627975) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa049c6b73000.stats=clear osc.lustre-OST0001-osc-ffffa049c6b73000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:33:01 (1786627981) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:33:13 (1786627993) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:33:20 (1786628000) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:33:23 (1786628003) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:33:31 (1786628011) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 09:33:40 (1786628020) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:33:48 (1786628028) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:33:56 (1786628036) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:34:04 (1786628044) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:34:13 (1786628053) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:34:19 (1786628059) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.726191 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719663 s, 14.6 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:34:27 (1786628067) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.922061 s, 19.3 MB/s PASS 22 (12s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:34:40 (1786628080) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:34:53 (1786628093) Stopping /mnt/lustre-ost1 (opts:) on oleg317-server oleg317-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa049c6b73000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa049c6b73000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg317-server: oleg317-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg317-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg317-server oleg317-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa049c6b73000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa049c6b73000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg317-server: oleg317-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg317-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:36:05 (1786628165) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.26413 s, 19.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg317-server oleg317-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa049c6b73000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa049c6b73000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg317-server: oleg317-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg317-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (37s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:36:42 (1786628202) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg317-server oleg317-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa049c6b73000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa049c6b73000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg317-server: oleg317-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg317-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg317-server oleg317-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa049c6b73000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa049c6b73000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg317-server: oleg317-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg317-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (70s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:37:52 (1786628272) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.49517 s, 21.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 13 09:37 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.792s user 0m0.002s sys 0m0.176s Stopping /mnt/lustre-ost1 (opts:) on oleg317-server oleg317-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa049c6b73000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa049c6b73000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.353s user 0m0.002s sys 0m0.160s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg317-server: oleg317-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg317-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg317-server oleg317-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa049c6b73000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa049c6b73000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.121s user 0m0.001s sys 0m0.189s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg317-server: oleg317-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg317-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg317-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa049c6b73000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa049c6b73000.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:39:21 (1786628361) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:39:25 (1786628365) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:39:28 (1786628368) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:39:31 (1786628371) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 09:39:37 (1786628377) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.520996 s, 24.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:39:48 (1786628388) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03514 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280248 s, 37.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:39:55 (1786628395) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:40:02 (1786628402) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000813845 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0020228 s, 506 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00205675 s, 498 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000945805 s, 1.1 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 09:40:11 (1786628411) create test files with size 15747798 .. 5bbc2686b2d9353a35b5fd01912c21d9 - 569300fdc90b12e92752f2b90e376c08 - 64ac95cc1e93017bb389ee78639551e3 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 09:40:37 (1786628437) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:40:49 (1786628449) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:40:55 (1786628455) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586751 s, 17.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xa6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205205 s, 51.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xa7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xa8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 09:41:03 (1786628463) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128661 s, 16.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.314662 s, 13.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167672 s, 12.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 09:41:15 (1786628475) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 09:41:18 (1786628478) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:41:21 (1786628481) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326656 s, 32.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 09:41:28 (1786628488) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.368954 s, 8.5 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:41:38 (1786628498) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xb2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:41:48 (1786628508) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.253752 s, 41.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:41:57 (1786628517) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa049d8473800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.264494 s, 39.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg317-client.virtnet /mnt/lustre (opts:) Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa049c4d06000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:42:10 (1786628530) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:42:47 (1786628567) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:43:22 (1786628602) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 09:43:26 (1786628606) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.275193 s, 11.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.175021 s, 18.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xbab6db29 (nosync,stale) Mirror 2: 0xbab6db29 (nosync,stale) ** verify mirror 2 doesn't change original checksum: f8bb41bbfabcd43c63ab7f365f902a24 - mirror 1 checksum: 66dc424eed4a61ce697df39bf5981d47 - mirror 2 checksum: f8bb41bbfabcd43c63ab7f365f902a24 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-13T09:43:29-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-13T09:43:29-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f8bb41bbfabcd43c63ab7f365f902a24 - mirror 1 checksum: 66dc424eed4a61ce697df39bf5981d47 - mirror 2 checksum: 66dc424eed4a61ce697df39bf5981d47 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-13T09:43:29-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-13T09:43:29-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdf:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:43:36 (1786628616) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334582 s, 31.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:43:42 (1786628622) Failing mds1 on oleg317-server Stopping /mnt/lustre-mds1 (opts:) on oleg317-server 09:43:54 (1786628634) shut down facet: mds1 facet_host: oleg317-server facet_failover_host: oleg317-server Failover mds1 to oleg317-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 09:44:15 (1786628655) targets are mounted 09:44:15 (1786628655) facet_failover done oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221904 s, 47.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe1:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:44:30 (1786628670) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0350387 s, 584 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0234075 s, 875 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:44:40 (1786628680) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.060265 s, 340 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0164457 s, 1.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:44:48 (1786628688) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0307317 s, 533 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:44:56 (1786628696) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0674057 s, 304 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:45:04 (1786628704) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0118868 s, 345 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0438039 s, 93.5 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:45:12 (1786628712) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:45:34 (1786628734) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:45:59 (1786628759) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:46:21 (1786628781) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:46:44 (1786628804) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 09:46:50 (1786628810) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57561 mirror split pid 57562 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57561 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57562 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 09:47:57 (1786628877) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 8879 000002 1786628879.435994 MIRROR_EXTEND to 2 mirrors 000003 1786628879.769013 write 0x037f01 thru 0x03ffff (0x80ff bytes) 000004 1786628879.842975 read 0x00191c thru 0x003a79 (0x215e bytes) 000005 1786628879.863811 mapread 0x015aa0 thru 0x01bc9d (0x61fe bytes) 000006 1786628879.899774 trunc from 0x027d34 to 0x02a002 (0x22cf bytes) 000007 1786628879.904734 write 0x00e9f7 thru 0x01b1e5 (0xc7ef bytes) 000008 1786628879.912560 mapwrite 0x03bf0d thru 0x03ffff (0x40f3 bytes) 000009 1786628879.976070 write 0x005b62 thru 0x00ef51 (0x93f0 bytes) 000010 1786628879.992239 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x666fdb64 Mirror 3: 0x666fdb64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x4cb4 000011 1786628881.261625 trunc from 0x040000 to 0x004cb3 (0x3b34c bytes) 000012 1786628881.362196 write 0x00703d thru 0x015fb2 (0xef76 bytes) zero_range to largest ever: 0x14a99 000013 1786628881.369400 trunc from 0x01337c to 0x014a98 (0x171d bytes) truncating to largest ever: 0x22ca5 000014 1786628881.420281 trunc from 0x015fb3 to 0x022ca4 (0xccf2 bytes) 000015 1786628881.485138 punch from 0x004726 to 0x00d866 (0x9141 bytes) 000016 1786628881.505019 mapwrite 0x03d88e thru 0x03ffff (0x2772 bytes) 000017 1786628881.614944 punch from 0x0162a4 to 0x0256ad (0xf40a bytes) 000018 1786628881.621902 read 0x028241 thru 0x02dd41 (0x5b01 bytes) 000019 1786628881.629934 write 0x00d86b thru 0x019b14 (0xc2aa bytes) 000021 1786628881.639951 punch from 0x01be49 to 0x02af3b (0xf0f3 bytes) 000022 1786628881.645078 punch from 0x034634 to 0x039d56 (0x5723 bytes) 000023 1786628881.649750 trunc from 0x040000 to 0x016856 (0x297a9 bytes) 000024 1786628881.671429 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16857): Mirror 3: 0x7f48f6eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16857): Mirror 3: 0x7f48f6eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16857: skipped 000025 1786628882.061401 fallocate 0x013cec thru 0x013d8c (0xa1 bytes) 000026 1786628882.073670 mapread 0x005c8f thru 0x00eb1f (0x8e91 bytes) 000027 1786628882.119621 punch from 0x00c897 to 0x016856 (0x9fc0 bytes) 000028 1786628882.135438 mapread 0x010b1a thru 0x013648 (0x2b2f bytes) fallocating to largest ever: 0x3b050 000029 1786628882.193430 fallocate 0x02d2a1 thru 0x03b04f (0xddaf bytes) 000030 1786628882.207380 trunc from 0x03b050 to 0x0120e8 (0x28f67 bytes) 000031 1786628882.242196 read 0x00d432 thru 0x0120e8 (0x4cb7 bytes) 000032 1786628882.250290 fallocate 0x028f25 thru 0x033ad1 (0xabad bytes) 000033 1786628882.265800 punch from 0x005ef9 to 0x006581 (0x689 bytes) 000034 1786628882.300510 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33ad2): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ad2): Mirror 3: 0x43d19c29 (stale) CRC-32 checksum value for chunk [0, 0x33ad2): Mirror 1: 0xd4e143c9 Mirror 4: 0xd4e143c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ad2: skipped 000035 1786628883.293909 trunc from 0x021912 to 0x027318 (0x5a07 bytes) 000036 1786628883.379440 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33ad2): Mirror 4: 0xd4e143c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ad2): Mirror 4: 0xd4e143c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ad2: skipped 000037 1786628883.729826 fallocate 0x005505 thru 0x008b28 (0x3624 bytes) 000038 1786628883.739155 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) 000039 1786628883.864020 write 0x019d8e thru 0x01bb7e (0x1df1 bytes) 000041 1786628883.876947 write 0x03f931 thru 0x03ffff (0x6cf bytes) fallocating to largest ever: 0x3dbfd 000042 1786628883.880032 fallocate 0x038d4c thru 0x03dbfc (0x4eb1 bytes) 000043 1786628883.885349 mapwrite 0x011b6f thru 0x01e830 (0xccc2 bytes) 000044 1786628883.933008 trunc from 0x00b321 to 0x015d47 (0xaa27 bytes) 000045 1786628883.939190 trunc from 0x040000 to 0x03399e (0xc661 bytes) 000046 1786628883.984909 mapread 0x000419 thru 0x0085e6 (0x81ce bytes) 000047 1786628884.046931 punch from 0x02e11d to 0x03399e (0x5882 bytes) 000049 1786628884.059955 punch from 0x022b1a to 0x02bd9e (0x9285 bytes) 000050 1786628884.064825 write 0x02279d thru 0x028fc8 (0x682c bytes) 000051 1786628884.075053 trunc from 0x00032f to 0x00ffe8 (0xfcba bytes) fallocating to largest ever: 0x40000 000052 1786628884.081951 fallocate 0x03bd13 thru 0x03ffff (0x42ed bytes) 000053 1786628884.090818 trunc from 0x03399f to 0x0072e1 (0x2c6bd bytes) 000055 1786628884.122767 fallocate 0x0132c7 thru 0x02309c (0xfdd6 bytes) 000056 1786628884.135020 mapwrite 0x01fa33 thru 0x027fab (0x8579 bytes) 000057 1786628884.196779 trunc from 0x002b8c to 0x003846 (0xcbb bytes) 000058 1786628884.221705 trunc from 0x027fac to 0x0108c1 (0x176ea bytes) 000059 1786628884.253541 mapread 0x00cb7f thru 0x00d9d8 (0xe5a bytes) 000060 1786628884.265371 mapwrite 0x03de66 thru 0x03ffff (0x219a bytes) 000061 1786628884.328038 mapread 0x02e81d thru 0x03a178 (0xb95c bytes) 000062 1786628884.359056 trunc from 0x040000 to 0x020c77 (0x1f388 bytes) 000063 1786628884.395648 read 0x01514b thru 0x020c77 (0xbb2d bytes) 000064 1786628884.422437 read 0x0057e0 thru 0x011098 (0xb8b9 bytes) 000065 1786628884.455729 write 0x033f8d thru 0x035095 (0x1109 bytes) 000066 1786628884.458599 read 0x023e96 thru 0x0293cb (0x5536 bytes) 000067 1786628884.481350 trunc from 0x035096 to 0x0274e5 (0xdbb0 bytes) 000069 1786628884.514732 punch from 0x013fa5 to 0x0187bc (0x4818 bytes) 000070 1786628884.519101 mapwrite 0x012eb1 thru 0x01dbcd (0xad1d bytes) 000071 1786628884.597325 trunc from 0x0274e6 to 0x000f15 (0x265d0 bytes) 000072 1786628884.625330 mapwrite 0x013c43 thru 0x01e0af (0xa46d bytes) 000073 1786628884.776827 punch from 0x0094d6 to 0x0180fb (0xec26 bytes) 000074 1786628884.790726 trunc from 0x00a186 to 0x00b657 (0x14d2 bytes) 000075 1786628884.797560 write 0x039ec6 thru 0x03c59c (0x26d7 bytes) 000076 1786628884.801002 mapread 0x0270d1 thru 0x03650e (0xf43e bytes) 000077 1786628884.904923 trunc from 0x030397 to 0x03819a (0x7e04 bytes) 000078 1786628884.931509 MIRROR_EXTEND to 2 mirrors 000079 1786628886.018267 MIRROR_EXTEND to 3 mirrors 000080 1786628886.989727 write 0x01f8eb thru 0x01f9c4 (0xda bytes) 000081 1786628887.059762 fallocate 0x03a2db thru 0x03db0c (0x3832 bytes) 000082 1786628887.071874 trunc from 0x03c59d to 0x00cd96 (0x2f806 bytes) 000083 1786628887.104924 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xcd97): Mirror 3: 0x33eee8a5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcd97): Mirror 3: 0x33eee8a5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcd97: skipped 000085 1786628887.388729 punch from 0x008693 to 0x00a0b2 (0x1a20 bytes) 000086 1786628887.394412 write 0x02a581 thru 0x0356d1 (0xb151 bytes) 000087 1786628887.400112 read 0x0088b3 thru 0x00c968 (0x40b6 bytes) 000088 1786628887.410045 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) 000089 1786628887.761129 MIRROR_EXTEND to 2 mirrors 000090 1786628888.745413 mapread 0x01c09e thru 0x027ef5 (0xbe58 bytes) 000092 1786628888.825877 MIRROR_RESYNC 2 mirrors 000093 1786628889.551607 trunc from 0x00faad to 0x011524 (0x1a78 bytes) 000094 1786628889.620327 trunc from 0x01b90c to 0x022fd9 (0x76ce bytes) 000095 1786628889.626335 mapread 0x002627 thru 0x0072a9 (0x4c83 bytes) 000096 1786628889.659056 mapwrite 0x029858 thru 0x02df29 (0x46d2 bytes) 000097 1786628889.710886 mapread 0x0262cf thru 0x0354cd (0xf1ff bytes) 000098 1786628889.731788 mapwrite 0x039c59 thru 0x03ffff (0x63a7 bytes) 000099 1786628889.847780 fallocate 0x026110 thru 0x027aaa (0x199b bytes) 000100 1786628889.858121 fallocate 0x027582 thru 0x02b783 (0x4202 bytes) 000101 1786628889.869239 mapread 0x03c10a thru 0x03ffff (0x3ef6 bytes) 000102 1786628889.870965 mapwrite 0x03d83f thru 0x03ffff (0x27c1 bytes) 000105 1786628889.913210 trunc from 0x01a4d9 to 0x01bcf2 (0x181a bytes) 000106 1786628889.947176 trunc from 0x00465c to 0x013d23 (0xf6c8 bytes) 000107 1786628889.960915 mapwrite 0x01f9a6 thru 0x02051d (0xb78 bytes) 000108 1786628890.048945 punch from 0x00b706 to 0x01056f (0x4e6a bytes) 000109 1786628890.057260 mapread 0x018afa thru 0x0219db (0x8ee2 bytes) 000111 1786628890.060704 write 0x004ee0 thru 0x00669b (0x17bc bytes) 000112 1786628890.074816 mapwrite 0x02945a thru 0x0360a2 (0xcc49 bytes) 000113 1786628890.098885 write 0x0369e0 thru 0x03eb46 (0x8167 bytes) 000114 1786628890.099938 mapwrite 0x0034e8 thru 0x004aa1 (0x15ba bytes) 000115 1786628890.145547 trunc from 0x040000 to 0x03daa7 (0x2558 bytes) 000117 1786628890.192962 read 0x007c3a thru 0x00acc4 (0x308b bytes) 000118 1786628890.210702 write 0x02641a thru 0x03070f (0xa2f6 bytes) 000119 1786628890.260936 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3daa8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3daa8): Mirror 2: 0x996ef055 (stale) CRC-32 checksum value for chunk [0, 0x3daa8): Mirror 1: 0x7d217cb9 Mirror 3: 0x7d217cb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3daa8: skipped 000120 1786628891.227862 fallocate 0x03dd93 thru 0x03fb41 (0x1daf bytes) 000121 1786628891.307684 trunc from 0x03fb42 to 0x01d30d (0x22834 bytes) 000122 1786628891.355256 read 0x00e745 thru 0x00fd1b (0x15d7 bytes) 000123 1786628891.360538 mapwrite 0x0076ff thru 0x00f393 (0x7c95 bytes) 000124 1786628891.404924 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d30e): Mirror 2: 0xe08d8f4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d30e): Mirror 2: 0xe08d8f4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d30e: skipped 000125 1786628891.712882 punch from 0x000db0 to 0x0075e9 (0x683a bytes) 000126 1786628891.719607 mapread 0x01cc59 thru 0x01d30d (0x6b5 bytes) 000128 1786628891.741002 punch from 0x00889b to 0x018774 (0xfeda bytes) 000130 1786628891.748975 read 0x01821b thru 0x01d30d (0x50f3 bytes) 000132 1786628891.760161 punch from 0x016126 to 0x01d30d (0x71e8 bytes) 000133 1786628891.767254 write 0x018b2c thru 0x01caa8 (0x3f7d bytes) 000134 1786628891.783403 trunc from 0x0102a4 to 0x01933c (0x9099 bytes) 000135 1786628891.801074 mapwrite 0x022400 thru 0x02c3c5 (0x9fc6 bytes) 000136 1786628891.953289 mapread 0x020cce thru 0x0221fb (0x152e bytes) 000138 1786628891.960144 trunc from 0x0218bc to 0x02c3c5 (0xab0a bytes) 000139 1786628891.982914 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) 000140 1786628892.364999 mapwrite 0x0139e0 thru 0x01f044 (0xb665 bytes) 000141 1786628892.477234 write 0x01dfc4 thru 0x01e3d1 (0x40e bytes) 000143 1786628892.515773 trunc from 0x02c3c6 to 0x039dc1 (0xd9fc bytes) 000146 1786628892.564701 mapread 0x0034f8 thru 0x007b19 (0x4622 bytes) 000147 1786628892.595238 punch from 0x02b5a0 to 0x034946 (0x93a7 bytes) 000148 1786628892.602269 write 0x019fd4 thru 0x0268eb (0xc918 bytes) 000149 1786628892.604172 read 0x0100d9 thru 0x01e9c7 (0xe8ef bytes) 000150 1786628892.609211 trunc from 0x008c2f to 0x013360 (0xa732 bytes) 000151 1786628892.617850 read 0x02a682 thru 0x034cd3 (0xa652 bytes) 000152 1786628892.653175 trunc from 0x000ab6 to 0x00347b (0x29c6 bytes) 000153 1786628892.665029 fallocate 0x021450 thru 0x02dc54 (0xc805 bytes) 000154 1786628892.674232 read 0x01cb41 thru 0x023ffb (0x74bb bytes) 000156 1786628892.685879 MIRROR_EXTEND to 2 mirrors 000157 1786628893.701528 read 0x01180e thru 0x016412 (0x4c05 bytes) 000158 1786628893.718933 MIRROR_RESYNC 2 mirrors 000160 1786628894.297653 mapwrite 0x034a16 thru 0x03ffff (0xb5ea bytes) 000161 1786628894.582957 write 0x03b095 thru 0x03ffff (0x4f6b bytes) 000162 1786628894.590499 fallocate 0x027abd thru 0x035512 (0xda56 bytes) 000163 1786628894.603789 trunc from 0x014acd to 0x018e8c (0x43c0 bytes) 000164 1786628894.614185 read 0x03a546 thru 0x03ffff (0x5aba bytes) 000165 1786628894.614900 write 0x034f83 thru 0x03ffff (0xb07d bytes) 000166 1786628894.616255 read 0x011757 thru 0x014fa0 (0x384a bytes) 000167 1786628894.652647 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) 000168 1786628894.890199 mapwrite 0x01f8fb thru 0x022969 (0x306f bytes) 000169 1786628894.941519 write 0x01fedf thru 0x02e1ad (0xe2cf bytes) 000170 1786628894.957997 read 0x028bfb thru 0x032125 (0x952b bytes) 000171 1786628894.970340 punch from 0x00ffa5 to 0x01c627 (0xc683 bytes) 000172 1786628894.991081 trunc from 0x002843 to 0x01197a (0xf138 bytes) 000174 1786628895.001118 fallocate 0x039e68 thru 0x03e5a7 (0x4740 bytes) 000175 1786628895.011720 read 0x01209e thru 0x021b92 (0xfaf5 bytes) 000177 1786628895.030810 punch from 0x03a720 to 0x03fc76 (0x5557 bytes) 000178 1786628895.041277 mapwrite 0x0220ac thru 0x02d85a (0xb7af bytes) 000179 1786628895.065175 mapread 0x019ab2 thru 0x021ac3 (0x8012 bytes) 000181 1786628895.069754 trunc from 0x027d84 to 0x030115 (0x8392 bytes) 000182 1786628895.076983 trunc from 0x040000 to 0x00cd7f (0x33280 bytes) 000183 1786628895.108202 fallocate 0x0173e2 thru 0x01cc1f (0x583e bytes) 000185 1786628895.115087 trunc from 0x0024d3 to 0x004f9f (0x2acd bytes) 000186 1786628895.119920 mapread 0x005ccd thru 0x00cd7f (0x70b3 bytes) 000187 1786628895.121147 mapwrite 0x011bb7 thru 0x020ccb (0xf115 bytes) 000188 1786628895.231589 mapwrite 0x039d20 thru 0x03d75e (0x3a3f bytes) 000189 1786628895.344922 mapwrite 0x0085ac thru 0x00f2dd (0x6d32 bytes) 000190 1786628895.392123 punch from 0x00a15c to 0x010499 (0x633e bytes) 000191 1786628895.403997 MIRROR_EXTEND to 2 mirrors 000192 1786628896.535184 trunc from 0x02a168 to 0x0315ae (0x7447 bytes) 000193 1786628896.567605 trunc from 0x03d75f to 0x018eea (0x24874 bytes) 000194 1786628896.608863 trunc from 0x009867 to 0x011104 (0x789e bytes) 000195 1786628896.616601 mapread 0x0062df thru 0x014e32 (0xeb54 bytes) 000197 1786628896.660313 trunc from 0x018eeb to 0x031542 (0x18658 bytes) 000198 1786628896.692277 trunc from 0x026f14 to 0x0280f6 (0x11e3 bytes) 000199 1786628896.706009 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31543): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31543): Mirror 2: 0x40df1e69 (stale) CRC-32 checksum value for chunk [0, 0x31543): Mirror 1: 0xd746c82d Mirror 3: 0xd746c82d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31543: skipped 000200 1786628897.877275 mapwrite 0x00dd97 thru 0x0128a6 (0x4b10 bytes) 000201 1786628898.062133 fallocate 0x0285b1 thru 0x028e69 (0x8b9 bytes) 000202 1786628898.087837 fallocate 0x027c3a thru 0x0307c0 (0x8b87 bytes) 000203 1786628898.096798 punch from 0x00b2c2 to 0x00f36c (0x40ab bytes) 000204 1786628898.105938 read 0x002b73 thru 0x012b03 (0xff91 bytes) 000205 1786628898.135575 trunc from 0x01a1b7 to 0x01f269 (0x50b3 bytes) 000206 1786628898.150501 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31543): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31543): Mirror 2: 0x40df1e69 (stale) Mirror 3: 0xd746c82d (stale) CRC-32 checksum value for chunk [0, 0x31543): Mirror 1: 0x8fb237c2 Mirror 4: 0x8fb237c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31543: skipped 000207 1786628899.256714 read 0x00856c thru 0x017a8a (0xf51f bytes) 000208 1786628899.281243 fallocate 0x027fb7 thru 0x028591 (0x5db bytes) 000209 1786628899.343985 trunc from 0x031543 to 0x0328e2 (0x13a0 bytes) 000210 1786628899.442131 read 0x020eee thru 0x02f41a (0xe52d bytes) 000211 1786628899.476774 read 0x0027af thru 0x00a951 (0x81a3 bytes) 000212 1786628899.477476 punch from 0x0238c4 to 0x023dc2 (0x4ff bytes) 000213 1786628899.489859 mapwrite 0x027261 thru 0x02dafb (0x689b bytes) 000214 1786628899.527467 MIRROR_RESYNC 4 mirrors 000215 1786628901.166018 mapread 0x013f0c thru 0x01ec21 (0xad16 bytes) 000216 1786628901.194026 mapread 0x00e6d6 thru 0x015390 (0x6cbb bytes) 000218 1786628901.195754 trunc from 0x0328e3 to 0x0253f0 (0xd4f2 bytes) 000219 1786628901.266676 fallocate 0x0064a4 thru 0x010746 (0xa2a3 bytes) 000221 1786628901.273856 trunc from 0x00f763 to 0x014a3b (0x52d9 bytes) 000223 1786628901.280655 trunc from 0x0253f1 to 0x03b5c8 (0x161d8 bytes) 000224 1786628901.306837 fallocate 0x00b274 thru 0x00c865 (0x15f2 bytes) 000225 1786628901.322451 MIRROR_RESYNC 4 mirrors 000226 1786628902.726363 mapread 0x013495 thru 0x01425c (0xdc8 bytes) 000227 1786628902.738133 fallocate 0x0006af thru 0x00fd7e (0xf6d0 bytes) 000228 1786628902.770756 trunc from 0x02728d to 0x0364af (0xf223 bytes) 000229 1786628902.776595 trunc from 0x038708 to 0x03b5c8 (0x2ec1 bytes) 000230 1786628902.780886 fallocate 0x02938a thru 0x038401 (0xf078 bytes) 000231 1786628902.786238 write 0x001840 thru 0x00e0b0 (0xc871 bytes) 000232 1786628902.802769 write 0x034563 thru 0x039ebc (0x595a bytes) 000233 1786628902.815602 read 0x004012 thru 0x00a104 (0x60f3 bytes) 000234 1786628902.816371 mapwrite 0x0177c3 thru 0x01e634 (0x6e72 bytes) 000235 1786628902.889724 read 0x022457 thru 0x031e98 (0xfa42 bytes) 000236 1786628902.905943 punch from 0x022a76 to 0x0272e6 (0x4871 bytes) 000238 1786628902.914848 mapread 0x01b0d5 thru 0x01fe6e (0x4d9a bytes) 000240 1786628902.921535 mapwrite 0x033155 thru 0x039a5f (0x690b bytes) 000241 1786628902.960202 write 0x037971 thru 0x03ffff (0x868f bytes) 000242 1786628902.961696 mapread 0x009601 thru 0x00ba7b (0x247b bytes) 000243 1786628902.968795 trunc from 0x040000 to 0x009da9 (0x36256 bytes) 000244 1786628903.003045 fallocate 0x002e2e thru 0x0097be (0x6991 bytes) 000245 1786628903.022494 read 0x00077e thru 0x009da9 (0x962c bytes) 000246 1786628903.024142 mapread 0x0003a1 thru 0x009da9 (0x9a09 bytes) 000247 1786628903.030600 trunc from 0x009daa to 0x02caf0 (0x22d47 bytes) 000248 1786628903.056724 trunc from 0x02caf1 to 0x029f68 (0x2b88 bytes) 000249 1786628903.087940 trunc from 0x029f69 to 0x0322a5 (0x833d bytes) 000250 1786628903.123231 trunc from 0x031276 to 0x0322a5 (0x1030 bytes) 000251 1786628903.132460 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x322a6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x322a6): Mirror 2: 0x5e148398 (stale) Mirror 3: 0x5e148398 (stale) Mirror 4: 0x5e148398 (stale) CRC-32 checksum value for chunk [0, 0x322a6): Mirror 1: 0xad552557 Mirror 5: 0xad552557 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x322a6: skipped 000252 1786628904.041764 punch from 0x02b34a to 0x02c32e (0xfe5 bytes) 000253 1786628904.093289 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x322a6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x322a6): Mirror 2: 0x5e148398 (stale) Mirror 3: 0x5e148398 (stale) Mirror 4: 0x5e148398 (stale) Mirror 5: 0xad552557 (stale) CRC-32 checksum value for chunk [0, 0x322a6): Mirror 1: 0xad552557 Mirror 6: 0xad552557 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x322a6: skipped 000254 1786628905.131470 fallocate 0x037c8d thru 0x0394b0 (0x1824 bytes) 000255 1786628905.180680 fallocate 0x004068 thru 0x007781 (0x371a bytes) 000256 1786628905.186792 trunc from 0x009613 to 0x00f098 (0x5a86 bytes) 000258 1786628905.202615 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x394b1): Mirror 2: 0x28a42ca9 (stale) Mirror 3: 0x28a42ca9 (stale) Mirror 4: 0x28a42ca9 (stale) Mirror 6: 0x9cc0381f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394b1): Mirror 2: 0x28a42ca9 (stale) Mirror 3: 0x28a42ca9 (stale) Mirror 4: 0x28a42ca9 (stale) Mirror 6: 0x9cc0381f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394b1: skipped 000259 1786628905.629148 MIRROR_RESYNC 5 mirrors 000260 1786628906.582957 trunc from 0x020f9d to 0x03069b (0xf6ff bytes) 000261 1786628906.629797 trunc from 0x025686 to 0x0287a8 (0x3123 bytes) 000262 1786628906.637095 fallocate 0x005e9f thru 0x0097ca (0x392c bytes) 000263 1786628906.643138 fallocate 0x000653 thru 0x004dea (0x4798 bytes) 000264 1786628906.649447 fallocate 0x03e088 thru 0x03ffff (0x1f78 bytes) 000265 1786628906.655387 fallocate 0x01224e thru 0x021814 (0xf5c7 bytes) 000266 1786628906.661215 fallocate 0x02009e thru 0x02b540 (0xb4a3 bytes) 000267 1786628906.672039 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2c57d210 (stale) Mirror 3: 0x2c57d210 (stale) Mirror 4: 0x2c57d210 (stale) Mirror 6: 0x2c57d210 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2c57d210 Mirror 7: 0x2c57d210 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000269 1786628907.610374 mapwrite 0x02229a thru 0x02230b (0x72 bytes) 000270 1786628907.721760 trunc from 0x040000 to 0x00e247 (0x31db8 bytes) 000271 1786628907.792939 fallocate 0x039fec thru 0x03ffff (0x6014 bytes) 000272 1786628907.804792 trunc from 0x03c43b to 0x03cac9 (0x68f bytes) 000274 1786628907.826294 punch from 0x01564c to 0x023ac6 (0xe47b bytes) 000275 1786628907.835469 punch from 0x02323e to 0x029a47 (0x680a bytes) 000276 1786628907.863349 mapwrite 0x0323c4 thru 0x03ff09 (0xdb46 bytes) 000277 1786628907.969850 fallocate 0x02f9c8 thru 0x03cb6b (0xd1a4 bytes) 000278 1786628907.977026 mapread 0x031b4d thru 0x03db3d (0xbff1 bytes) 000279 1786628907.979150 fallocate 0x0300e5 thru 0x0312c7 (0x11e3 bytes) 000280 1786628907.984978 mapwrite 0x0210c2 thru 0x028ba3 (0x7ae2 bytes) 000281 1786628908.034018 mapwrite 0x034833 thru 0x038bfc (0x43ca bytes) 000282 1786628908.052462 mapwrite 0x021525 thru 0x022878 (0x1354 bytes) 000283 1786628908.067216 trunc from 0x01851e to 0x022527 (0xa00a bytes) 000284 1786628908.073650 write 0x02eafa thru 0x035c11 (0x7118 bytes) 000285 1786628908.076443 mapwrite 0x016cf3 thru 0x01eeb8 (0x81c6 bytes) 000286 1786628908.113040 MIRROR_RESYNC 6 mirrors 000287 1786628910.336007 punch from 0x010a4b to 0x014470 (0x3a26 bytes) 000288 1786628910.375164 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52421008 (stale) Mirror 3: 0x52421008 (stale) Mirror 4: 0x52421008 (stale) Mirror 6: 0x52421008 (stale) Mirror 7: 0x52421008 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x52421008 Mirror 8: 0x52421008 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000289 1786628911.580263 fallocate 0x02c8a2 thru 0x02d489 (0xbe8 bytes) 000290 1786628911.662018 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52421008 (stale) Mirror 3: 0x52421008 (stale) Mirror 6: 0x52421008 (stale) Mirror 7: 0x52421008 (stale) Mirror 8: 0x52421008 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52421008 (stale) Mirror 3: 0x52421008 (stale) Mirror 6: 0x52421008 (stale) Mirror 7: 0x52421008 (stale) Mirror 8: 0x52421008 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000291 1786628912.287092 trunc from 0x0104b2 to 0x017845 (0x7394 bytes) 000292 1786628912.298066 mapread 0x007362 thru 0x00ac42 (0x38e1 bytes) 000293 1786628912.361151 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52421008 (stale) Mirror 3: 0x52421008 (stale) Mirror 6: 0x52421008 (stale) Mirror 7: 0x52421008 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52421008 (stale) Mirror 3: 0x52421008 (stale) Mirror 6: 0x52421008 (stale) Mirror 7: 0x52421008 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000294 1786628912.963725 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52421008 (stale) Mirror 3: 0x52421008 (stale) Mirror 6: 0x52421008 (stale) Mirror 7: 0x52421008 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5e15644c Mirror 8: 0x5e15644c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000295 1786628914.222246 trunc from 0x040000 to 0x01b868 (0x24797 bytes) 000296 1786628914.279654 trunc from 0x01b869 to 0x021d09 (0x64a1 bytes) 000297 1786628914.316977 mapwrite 0x00513d thru 0x013df7 (0xecbb bytes) 000298 1786628914.441458 trunc from 0x021d0a to 0x01e957 (0x33b2 bytes) 000299 1786628914.475592 fallocate 0x0218e8 thru 0x02a40e (0x8b27 bytes) 000300 1786628914.492762 fallocate 0x01be1a thru 0x0237f3 (0x79da bytes) 000301 1786628914.510436 mapread 0x01864d thru 0x01e957 (0x630b bytes) 000303 1786628914.533862 write 0x02d0c3 thru 0x030f3b (0x3e79 bytes) 000304 1786628914.535744 punch from 0x0290ab to 0x02a472 (0x13c8 bytes) 000306 1786628914.548176 trunc from 0x01a117 to 0x01f3b7 (0x52a1 bytes) 000307 1786628914.571010 punch from 0x00a09c to 0x014c7f (0xabe4 bytes) 000308 1786628914.603635 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30f3c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f3c): Mirror 2: 0x18746789 (stale) Mirror 3: 0x18746789 (stale) Mirror 6: 0x18746789 (stale) Mirror 7: 0x18746789 (stale) Mirror 8: 0x2d15a6d7 (stale) CRC-32 checksum value for chunk [0, 0x30f3c): Mirror 1: 0x989c8d7d Mirror 9: 0x989c8d7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f3c: skipped 000309 1786628915.753164 mapwrite 0x037e83 thru 0x03f6a6 (0x7824 bytes) 000310 1786628915.871832 mapread 0x0142fc thru 0x0157e6 (0x14eb bytes) 000312 1786628915.904181 read 0x02dc7f thru 0x02ed2c (0x10ae bytes) 000314 1786628915.910802 trunc from 0x03f6a7 to 0x02a034 (0x15672 bytes) 000316 1786628915.955866 read 0x028d1a thru 0x02a034 (0x131b bytes) 000317 1786628915.972196 trunc from 0x00b36e to 0x019187 (0xde1a bytes) 000319 1786628915.981002 read 0x00aab7 thru 0x014044 (0x958e bytes) 000320 1786628915.998497 write 0x013e38 thru 0x01eaff (0xacc8 bytes) 000321 1786628916.002487 mapread 0x00aebb thru 0x01696d (0xbab3 bytes) 000322 1786628916.010188 fallocate 0x0014b7 thru 0x001a1d (0x567 bytes) 000323 1786628916.028064 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a035): Mirror 2: 0xea8aae3 (stale) Mirror 3: 0xea8aae3 (stale) Mirror 6: 0xea8aae3 (stale) Mirror 8: 0xd3593be3 (stale) Mirror 9: 0xc02bb33f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a035): Mirror 2: 0xea8aae3 (stale) Mirror 3: 0xea8aae3 (stale) Mirror 6: 0xea8aae3 (stale) Mirror 8: 0xd3593be3 (stale) Mirror 9: 0xc02bb33f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a035: skipped 000324 1786628916.517145 trunc from 0x00423f to 0x004ea6 (0xc68 bytes) 000325 1786628916.522290 punch from 0x01df82 to 0x023b62 (0x5be1 bytes) 000326 1786628916.529270 read 0x0072c5 thru 0x00c6a9 (0x53e5 bytes) 000327 1786628916.540967 trunc from 0x01ed1b to 0x02a034 (0xb31a bytes) 000328 1786628916.549847 mapwrite 0x025e45 thru 0x026f8d (0x1149 bytes) 000329 1786628916.593091 fallocate 0x01309e thru 0x0217c7 (0xe72a bytes) 000330 1786628916.603729 trunc from 0x02a035 to 0x001161 (0x28ed3 bytes) 000331 1786628916.644382 trunc from 0x001162 to 0x0104bf (0xf35e bytes) 000332 1786628916.664075 read 0x00f23d thru 0x0104bf (0x1283 bytes) 000333 1786628916.677901 trunc from 0x0104c0 to 0x00a72b (0x5d94 bytes) 000334 1786628916.703550 trunc from 0x00a72c to 0x03bc7a (0x3154f bytes) 000335 1786628916.754126 trunc from 0x03bc7b to 0x0395c9 (0x26b1 bytes) 000336 1786628916.788020 fallocate 0x03a818 thru 0x03ffff (0x57e8 bytes) 000337 1786628916.807270 trunc from 0x03c3bb to 0x03ffff (0x3c45 bytes) 000338 1786628916.818821 trunc from 0x040000 to 0x036605 (0x99fa bytes) 000339 1786628916.848944 mapwrite 0x02a97b thru 0x036237 (0xb8bd bytes) 000340 1786628916.924360 punch from 0x00aa33 to 0x019712 (0xece0 bytes) 000341 1786628916.950284 read 0x03248b thru 0x036605 (0x417b bytes) 000342 1786628916.951183 punch from 0x012e1c to 0x0180c2 (0x52a7 bytes) 000343 1786628916.963842 write 0x00037f thru 0x00ab9a (0xa81c bytes) 000344 1786628916.979529 fallocate 0x03508e thru 0x0360d4 (0x1047 bytes) 000346 1786628916.985267 trunc from 0x018053 to 0x0182a3 (0x251 bytes) 000347 1786628916.990706 read 0x022b2b thru 0x02e6e9 (0xbbbf bytes) 000348 1786628916.994047 fallocate 0x03c910 thru 0x03ffff (0x36f0 bytes) 000349 1786628917.016606 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52421008 (stale) Mirror 3: 0x52421008 (stale) Mirror 6: 0x52421008 (stale) Mirror 9: 0x5f1285d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52421008 (stale) Mirror 3: 0x52421008 (stale) Mirror 6: 0x52421008 (stale) Mirror 9: 0x5f1285d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000350 1786628917.614869 write 0x02f30d thru 0x03ca36 (0xd72a bytes) 000351 1786628917.646079 mapwrite 0x004e70 thru 0x0135e2 (0xe773 bytes) 000352 1786628917.716888 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x52421008 (stale) Mirror 3: 0x52421008 (stale) Mirror 6: 0x52421008 (stale) Mirror 9: 0x5f1285d9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbcb3cf83 Mirror 10: 0xbcb3cf83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000353 1786628918.801932 mapread 0x03a337 thru 0x03ffff (0x5cc9 bytes) 000354 1786628918.859843 mapwrite 0x00517b thru 0x01516f (0xfff5 bytes) 000355 1786628919.120520 fallocate 0x039a0e thru 0x03b1ff (0x17f2 bytes) 000356 1786628919.144766 fallocate 0x02cb30 thru 0x03b425 (0xe8f6 bytes) 000357 1786628919.162854 trunc from 0x040000 to 0x031257 (0xeda8 bytes) 000358 1786628919.222659 trunc from 0x0288da to 0x029a51 (0x1178 bytes) 000359 1786628919.231963 mapread 0x00d722 thru 0x01702a (0x9909 bytes) 000360 1786628919.250020 MIRROR_RESYNC 6 mirrors 000361 1786628920.657008 punch from 0x0296e0 to 0x02c7c4 (0x30e5 bytes) 000362 1786628920.687793 fallocate 0x0157af thru 0x02374c (0xdf9e bytes) 000363 1786628920.692670 punch from 0x00072b to 0x008b4e (0x8424 bytes) 000364 1786628920.701524 MIRROR_RESYNC 6 mirrors 000365 1786628922.372526 read 0x011bab thru 0x01e6da (0xcb30 bytes) 000366 1786628922.381872 read 0x01eaec thru 0x01ece7 (0x1fc bytes) 000368 1786628922.382713 read 0x00fa9f thru 0x00fabf (0x21 bytes) 000369 1786628922.396191 write 0x02ae99 thru 0x032dfb (0x7f63 bytes) 000370 1786628922.430976 trunc from 0x0036ae to 0x007224 (0x3b77 bytes) 000371 1786628922.437802 write 0x01be1b thru 0x01e478 (0x265e bytes) 000372 1786628922.447685 mapread 0x01341b thru 0x0199af (0x6595 bytes) 000373 1786628922.482105 trunc from 0x019612 to 0x01de8e (0x487d bytes) 000374 1786628922.513495 mapwrite 0x036e97 thru 0x039f0f (0x3079 bytes) 000375 1786628922.600784 punch from 0x00b534 to 0x016b52 (0xb61f bytes) 000376 1786628922.611951 mapwrite 0x012c11 thru 0x01f6c3 (0xcab3 bytes) 000377 1786628922.697570 write 0x02a69d thru 0x032e0b (0x876f bytes) 000378 1786628922.698998 mapwrite 0x032b0b thru 0x03c7d2 (0x9cc8 bytes) 000380 1786628922.765801 trunc from 0x03c7d3 to 0x029998 (0x12e3a bytes) 000381 1786628922.790041 trunc from 0x00caeb to 0x016b6f (0xa085 bytes) 000382 1786628922.806206 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29999): Mirror 2: 0xe5cda230 (stale) Mirror 3: 0xe5cda230 (stale) Mirror 9: 0xe5cda230 (stale) Mirror 10: 0xe5cda230 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29999): Mirror 2: 0xe5cda230 (stale) Mirror 3: 0xe5cda230 (stale) Mirror 9: 0xe5cda230 (stale) Mirror 10: 0xe5cda230 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29999: skipped 000383 1786628923.255789 mapwrite 0x01123d thru 0x012a10 (0x17d4 bytes) 000384 1786628923.305342 punch from 0x01e0f8 to 0x025c86 (0x7b8f bytes) 000385 1786628923.310615 punch from 0x01b7e6 to 0x0252c9 (0x9ae4 bytes) 000386 1786628923.316079 write 0x014455 thru 0x01c7d7 (0x8383 bytes) 000387 1786628923.333937 trunc from 0x0154f4 to 0x016b14 (0x1621 bytes) 000388 1786628923.352017 MIRROR_RESYNC 5 mirrors 000389 1786628924.902572 punch from 0x025903 to 0x029998 (0x4096 bytes) 000390 1786628925.013635 mapread 0x00ee11 thru 0x010428 (0x1618 bytes) 000391 1786628925.064196 fallocate 0x024c61 thru 0x03198d (0xcd2d bytes) 000392 1786628925.085416 read 0x00276a thru 0x0114dd (0xed74 bytes) 000393 1786628925.101663 mapread 0x01b431 thru 0x024e6f (0x9a3f bytes) 000394 1786628925.175507 read 0x01441d thru 0x0151eb (0xdcf bytes) 000395 1786628925.176364 read 0x0189bd thru 0x01da60 (0x50a4 bytes) 000396 1786628925.177594 trunc from 0x029999 to 0x029e79 (0x4e1 bytes) 000397 1786628925.218534 mapwrite 0x003a03 thru 0x011f72 (0xe570 bytes) 000398 1786628925.282097 fallocate 0x0317c9 thru 0x03b7e8 (0xa020 bytes) 000399 1786628925.296596 punch from 0x00cadb to 0x0196c6 (0xcbec bytes) 000400 1786628925.311536 trunc from 0x01407f to 0x0155ca (0x154c bytes) 000402 1786628925.324637 trunc from 0x029e7a to 0x0135c4 (0x168b5 bytes) 000403 1786628925.342377 trunc from 0x0135c5 to 0x0107f3 (0x2dd1 bytes) 000404 1786628925.361078 fallocate 0x0319fe thru 0x035461 (0x3a64 bytes) 000405 1786628925.380602 mapread 0x005e96 thru 0x0067d8 (0x943 bytes) 000407 1786628925.388727 read 0x0068eb thru 0x00f52d (0x8c43 bytes) 000408 1786628925.403058 trunc from 0x00d4c1 to 0x0107f3 (0x3333 bytes) 000409 1786628925.413947 mapread 0x007ef7 thru 0x007f31 (0x3b bytes) 000410 1786628925.416427 trunc from 0x00d979 to 0x0107f3 (0x2e7b bytes) 000411 1786628925.424656 punch from 0x0008d5 to 0x00f1df (0xe90b bytes) 000413 1786628925.433677 mapread 0x00c6de thru 0x00fa26 (0x3349 bytes) 000414 1786628925.454980 trunc from 0x003268 to 0x004af8 (0x1891 bytes) 000415 1786628925.460505 mapwrite 0x010b27 thru 0x011203 (0x6dd bytes) 000416 1786628925.501090 write 0x034356 thru 0x03860b (0x42b6 bytes) 000417 1786628925.503722 trunc from 0x03860c to 0x011f3b (0x266d0 bytes) 000418 1786628925.550745 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11f3c): Mirror 2: 0xe8d57f05 (stale) Mirror 3: 0xe8d57f05 (stale) Mirror 10: 0xe8d57f05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11f3c): Mirror 2: 0xe8d57f05 (stale) Mirror 3: 0xe8d57f05 (stale) Mirror 10: 0xe8d57f05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11f3c: skipped 000419 1786628926.051759 punch from 0x009b91 to 0x011f3b (0x83ab bytes) 000420 1786628926.071959 trunc from 0x0008e0 to 0x005702 (0x4e23 bytes) 000421 1786628926.080016 punch from 0x01150b to 0x011f3b (0xa31 bytes) 000422 1786628926.087784 fallocate 0x008b88 thru 0x012cab (0xa124 bytes) 000423 1786628926.094009 punch from 0x00058a to 0x00ef3c (0xe9b3 bytes) 000424 1786628926.104380 punch from 0x003017 to 0x00cec2 (0x9eac bytes) 000425 1786628926.109429 punch from 0x005f56 to 0x011f3b (0xbfe6 bytes) 000426 1786628926.115746 trunc from 0x011f3c to 0x0232a5 (0x1136a bytes) 000428 1786628926.158865 trunc from 0x0232a6 to 0x0131f0 (0x100b5 bytes) 000429 1786628926.183994 punch from 0x0043dd to 0x00f5e7 (0xb20b bytes) 000430 1786628926.189732 mapread 0x008004 thru 0x00bdc6 (0x3dc3 bytes) 000431 1786628926.222403 read 0x00ba91 thru 0x011d6a (0x62da bytes) 000432 1786628926.223061 write 0x01bb05 thru 0x029f9c (0xe498 bytes) 000433 1786628926.227402 read 0x01e4de thru 0x021d77 (0x389a bytes) 000434 1786628926.229048 read 0x012213 thru 0x016b00 (0x48ee bytes) 000435 1786628926.240668 mapread 0x028e00 thru 0x029f9c (0x119d bytes) 000436 1786628926.241793 mapwrite 0x029e30 thru 0x03510c (0xb2dd bytes) 000437 1786628926.362174 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3510d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3510d): Mirror 2: 0x1c05ff9 (stale) Mirror 3: 0x1c05ff9 (stale) Mirror 10: 0x1c05ff9 (stale) CRC-32 checksum value for chunk [0, 0x3510d): Mirror 1: 0xeb693f0 Mirror 11: 0xeb693f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3510d: skipped 000438 1786628927.665171 fallocate 0x0308fb thru 0x03ccdd (0xc3e3 bytes) 000439 1786628927.733211 read 0x007783 thru 0x00d72b (0x5fa9 bytes) 000441 1786628927.748684 trunc from 0x0310a8 to 0x03510c (0x4065 bytes) 000442 1786628927.756811 mapwrite 0x018719 thru 0x026c44 (0xe52c bytes) 000443 1786628927.816275 MIRROR_RESYNC 5 mirrors 000444 1786628929.364068 write 0x004a27 thru 0x00ab4c (0x6126 bytes) 000445 1786628929.434574 MIRROR_RESYNC 5 mirrors 000446 1786628931.537564 fallocate 0x00c2e2 thru 0x018a52 (0xc771 bytes) 000447 1786628931.611088 trunc from 0x02ab09 to 0x02f234 (0x472c bytes) 000448 1786628931.624193 read 0x013129 thru 0x01c12f (0x9007 bytes) 000449 1786628931.644275 punch from 0x027bbf to 0x031749 (0x9b8b bytes) 000450 1786628931.651224 read 0x01f539 thru 0x023893 (0x435b bytes) 000451 1786628931.662956 mapread 0x0020f0 thru 0x006017 (0x3f28 bytes) 000452 1786628931.664947 trunc from 0x03510d to 0x030ff3 (0x4119 bytes) 000453 1786628931.718884 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30ff4): Mirror 3: 0x5e7607f1 (stale) Mirror 10: 0x5e7607f1 (stale) Mirror 11: 0x5e7607f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ff4): Mirror 3: 0x5e7607f1 (stale) Mirror 10: 0x5e7607f1 (stale) Mirror 11: 0x5e7607f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ff4: skipped 000454 1786628932.240755 punch from 0x000e82 to 0x004e23 (0x3fa2 bytes) 000455 1786628932.259043 trunc from 0x030ff4 to 0x00f33d (0x21cb6 bytes) 000456 1786628932.298675 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf33e): Mirror 10: 0x37966a4d (stale) Mirror 11: 0x37966a4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf33e): Mirror 10: 0x37966a4d (stale) Mirror 11: 0x37966a4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf33e: skipped 000457 1786628932.576797 mapread 0x002aee thru 0x00f33d (0xc850 bytes) 000458 1786628932.608869 mapwrite 0x009938 thru 0x011570 (0x7c39 bytes) 000459 1786628932.680275 trunc from 0x011571 to 0x03ea0d (0x2d49d bytes) 000460 1786628932.724515 trunc from 0x0092fa to 0x011b00 (0x8807 bytes) 000461 1786628932.747020 mapwrite 0x028e0a thru 0x02eb88 (0x5d7f bytes) 000462 1786628932.779938 write 0x029362 thru 0x02f5e8 (0x6287 bytes) 000463 1786628932.781568 trunc from 0x03ea0e to 0x035b61 (0x8eac bytes) 000464 1786628932.801226 read 0x026759 thru 0x02f71f (0x8fc7 bytes) 000465 1786628932.802036 mapread 0x01a9b9 thru 0x01b971 (0xfb9 bytes) 000466 1786628932.832544 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35b62): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b62): Mirror 10: 0x2ecaad3d (stale) Mirror 11: 0x2ecaad3d (stale) CRC-32 checksum value for chunk [0, 0x35b62): Mirror 1: 0xfcd53188 Mirror 12: 0xfcd53188 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b62: skipped 000467 1786628933.936060 mapread 0x00d30e thru 0x01ab18 (0xd80b bytes) 000468 1786628933.990382 read 0x01fadf thru 0x02a0e5 (0xa607 bytes) 000469 1786628934.005802 trunc from 0x035b62 to 0x01fe66 (0x15cfb bytes) 000470 1786628934.101767 trunc from 0x01fd01 to 0x01fe66 (0x166 bytes) 000471 1786628934.122483 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fe67): Mirror 11: 0xccab97fa (stale) Mirror 12: 0xfbb4f1f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fe67): Mirror 11: 0xccab97fa (stale) Mirror 12: 0xfbb4f1f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fe67: skipped 000472 1786628934.532931 mapread 0x00b8af thru 0x0136f2 (0x7e44 bytes) 000473 1786628934.554403 write 0x01aa3d thru 0x02084a (0x5e0e bytes) 000474 1786628934.565587 fallocate 0x01a47e thru 0x02310c (0x8c8f bytes) 000475 1786628934.570768 read 0x020824 thru 0x02310c (0x28e9 bytes) 000476 1786628934.578243 write 0x02ecb0 thru 0x039195 (0xa4e6 bytes) 000477 1786628934.580909 mapwrite 0x0344d1 thru 0x03b069 (0x6b99 bytes) 000478 1786628934.645056 read 0x0080d1 thru 0x013c26 (0xbb56 bytes) 000479 1786628934.646925 punch from 0x0383fc to 0x03b069 (0x2c6e bytes) 000480 1786628934.654254 trunc from 0x00121b to 0x00fbdd (0xe9c3 bytes) 000481 1786628934.659704 write 0x002277 thru 0x003109 (0xe93 bytes) 000482 1786628934.687512 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b06a): Mirror 11: 0xfd7f5d88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b06a): Mirror 11: 0xfd7f5d88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b06a: skipped 000483 1786628935.015845 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b06a): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b06a): Mirror 11: 0xfd7f5d88 (stale) CRC-32 checksum value for chunk [0, 0x3b06a): Mirror 1: 0xa321dd83 Mirror 12: 0xa321dd83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b06a: skipped 000484 1786628936.322909 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b06a): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b06a): Mirror 11: 0xfd7f5d88 (stale) CRC-32 checksum value for chunk [0, 0x3b06a): Mirror 1: 0xa321dd83 Mirror 12: 0xa321dd83 Mirror 13: 0xa321dd83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b06a: skipped 000485 1786628937.557892 write 0x03acaa thru 0x03ffff (0x5356 bytes) 000487 1786628937.620219 trunc from 0x040000 to 0x0075b9 (0x38a46 bytes) 000488 1786628937.663862 write 0x0064fa thru 0x00d668 (0x716f bytes) 000489 1786628937.693910 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd669): Mirror 11: 0xbeb8c40e (stale) Mirror 13: 0xa1d92150 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd669): Mirror 11: 0xbeb8c40e (stale) Mirror 13: 0xa1d92150 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd669: skipped 000490 1786628938.035339 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd669): Mirror 11: 0xbeb8c40e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd669): Mirror 11: 0xbeb8c40e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd669: skipped 000491 1786628938.525504 punch from 0x00a1e0 to 0x00d668 (0x3489 bytes) 000493 1786628938.537413 trunc from 0x0038bd to 0x00d668 (0x9dac bytes) 000494 1786628938.546600 read 0x007be6 thru 0x00d668 (0x5a83 bytes) 000495 1786628938.565368 punch from 0x00c802 to 0x00d668 (0xe67 bytes) 000497 1786628938.578523 fallocate 0x01c106 thru 0x020c03 (0x4afe bytes) 000498 1786628938.587758 punch from 0x009b57 to 0x00d668 (0x3b12 bytes) 000499 1786628938.595453 mapwrite 0x016e76 thru 0x01a813 (0x399e bytes) 000500 1786628938.716675 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) 000502 1786628938.862814 trunc from 0x013f2c to 0x017d8f (0x3e64 bytes) 000503 1786628938.871535 trunc from 0x018e69 to 0x01a813 (0x19ab bytes) 000504 1786628938.878248 read 0x00e57e thru 0x01a813 (0xc296 bytes) 000505 1786628938.889743 trunc from 0x01a814 to 0x00dd00 (0xcb13 bytes) 000506 1786628938.921894 mapwrite 0x0142b5 thru 0x01b218 (0x6f64 bytes) 000507 1786628939.019620 mapwrite 0x00793e thru 0x00983d (0x1f00 bytes) 000509 1786628939.058669 trunc from 0x00ae8a to 0x00ccab (0x1e22 bytes) 000510 1786628939.065938 write 0x0092cf thru 0x0125e7 (0x9319 bytes) 000511 1786628939.071490 trunc from 0x01b219 to 0x01a394 (0xe84 bytes) 000512 1786628939.102920 fallocate 0x0319a3 thru 0x03ce43 (0xb4a1 bytes) 000513 1786628939.110494 trunc from 0x03ce44 to 0x02a1f6 (0x12c4d bytes) 000514 1786628939.124567 write 0x015ac8 thru 0x023abf (0xdff8 bytes) 000515 1786628939.134850 trunc from 0x02a1f7 to 0x019acc (0x1072a bytes) 000516 1786628939.158665 mapread 0x011c1f thru 0x016f30 (0x5312 bytes) 000517 1786628939.160014 read 0x000a0b thru 0x001a1d (0x1013 bytes) 000519 1786628939.168411 read 0x00758d thru 0x00ce0e (0x5882 bytes) 000520 1786628939.169505 mapread 0x013492 thru 0x0141a3 (0xd12 bytes) 000521 1786628939.172223 mapwrite 0x0207db thru 0x022e57 (0x267d bytes) 000522 1786628939.214998 write 0x033467 thru 0x03371b (0x2b5 bytes) 000523 1786628939.217561 mapwrite 0x005925 thru 0x009f1f (0x45fb bytes) 000526 1786628939.243648 punch from 0x0266e2 to 0x02b00e (0x492d bytes) 000528 1786628939.249820 write 0x036d24 thru 0x039d7a (0x3057 bytes) 000529 1786628939.251932 mapwrite 0x011470 thru 0x0196ea (0x827b bytes) 000530 1786628939.274486 read 0x00e723 thru 0x01d8c0 (0xf19e bytes) 000531 1786628939.276265 mapread 0x02a306 thru 0x031c62 (0x795d bytes) 000532 1786628939.290555 fallocate 0x03ce2d thru 0x03ffff (0x31d3 bytes) 000535 1786628939.302604 read 0x02c681 thru 0x02c845 (0x1c5 bytes) 000537 1786628939.305379 read 0x01887d thru 0x023347 (0xaacb bytes) 000538 1786628939.306161 trunc from 0x000692 to 0x001d7b (0x16ea bytes) 000540 1786628939.313558 write 0x026486 thru 0x02ffa2 (0x9b1d bytes) 000541 1786628939.315810 read 0x017c51 thru 0x020d84 (0x9134 bytes) 000542 1786628939.316676 mapwrite 0x02ed9e thru 0x033937 (0x4b9a bytes) 000543 1786628939.334251 trunc from 0x00a4aa to 0x00f8d3 (0x542a bytes) 000544 1786628939.345400 mapwrite 0x01232b thru 0x017af9 (0x57cf bytes) 000545 1786628939.374702 punch from 0x0150b3 to 0x0217d8 (0xc726 bytes) 000546 1786628939.390271 trunc from 0x039d7b to 0x00b00c (0x2ed6e bytes) 000547 1786628939.421373 write 0x03c6d8 thru 0x03d9a2 (0x12cb bytes) 000548 1786628939.423590 punch from 0x022e2b to 0x027d96 (0x4f6c bytes) 000549 1786628939.429371 trunc from 0x024817 to 0x02e6e9 (0x9ed3 bytes) 000550 1786628939.437387 mapwrite 0x034f7a thru 0x03f4fa (0xa581 bytes) 000551 1786628939.490462 read 0x032275 thru 0x03903e (0x6dca bytes) 000552 1786628939.491364 mapwrite 0x022e3d thru 0x03155f (0xe723 bytes) 000553 1786628939.540918 punch from 0x03dcc9 to 0x03f4fa (0x1832 bytes) 000554 1786628939.553049 read 0x007735 thru 0x01657e (0xee4a bytes) 000555 1786628939.554107 read 0x016628 thru 0x024ea1 (0xe87a bytes) 000556 1786628939.555589 mapread 0x012f87 thru 0x018e0c (0x5e86 bytes) 000557 1786628939.560482 punch from 0x03367f to 0x03c073 (0x89f5 bytes) 000558 1786628939.566896 trunc from 0x03f4fb to 0x02bc06 (0x138f4 bytes) 000560 1786628939.591549 write 0x0315e7 thru 0x03d74a (0xc164 bytes) 000562 1786628939.594260 trunc from 0x03cf01 to 0x03d74a (0x84a bytes) 000563 1786628939.612662 punch from 0x02f69e to 0x03b7a7 (0xc10a bytes) 000565 1786628939.620697 trunc from 0x03d74b to 0x02e4e1 (0xf269 bytes) 000566 1786628939.646024 mapwrite 0x000fa6 thru 0x002045 (0x10a0 bytes) 000567 1786628939.667706 read 0x00a7eb thru 0x00e338 (0x3b4e bytes) 000568 1786628939.669196 trunc from 0x00598a to 0x00dbfd (0x8274 bytes) 000570 1786628939.678027 mapread 0x02cba1 thru 0x02e4e1 (0x1941 bytes) 000571 1786628939.692791 mapread 0x0114bd thru 0x0149fe (0x3542 bytes) 000572 1786628939.694257 punch from 0x00cc1b to 0x0176d9 (0xaabf bytes) 000573 1786628939.703838 mapread 0x015b3c thru 0x01cb9f (0x7064 bytes) 000574 1786628939.720644 trunc from 0x00f3a7 to 0x00f9bd (0x617 bytes) 000575 1786628939.727745 mapread 0x001b7c thru 0x0029a3 (0xe28 bytes) 000576 1786628939.730515 mapread 0x0152de thru 0x02065b (0xb37e bytes) 000577 1786628939.732971 mapread 0x00f236 thru 0x017b03 (0x88ce bytes) 000578 1786628939.772768 mapwrite 0x013c8c thru 0x01759a (0x390f bytes) 000579 1786628939.789983 trunc from 0x02e4e2 to 0x0088ba (0x25c27 bytes) 000580 1786628939.804641 read 0x0038bf thru 0x0086ec (0x4e2e bytes) 000581 1786628939.807900 trunc from 0x003a19 to 0x008156 (0x473e bytes) 000582 1786628939.814185 mapwrite 0x0281b0 thru 0x03096e (0x87bf bytes) 000583 1786628939.860830 mapwrite 0x00aa24 thru 0x0140d9 (0x96b6 bytes) 000584 1786628939.880939 mapwrite 0x01a85d thru 0x027624 (0xcdc8 bytes) 000586 1786628939.912286 trunc from 0x01beb7 to 0x024fd6 (0x9120 bytes) 000587 1786628939.917564 mapwrite 0x0099bf thru 0x00a258 (0x89a bytes) 000588 1786628939.929142 mapwrite 0x039683 thru 0x03a1af (0xb2d bytes) 000589 1786628939.965812 mapwrite 0x00e247 thru 0x01a549 (0xc303 bytes) 000590 1786628939.993939 punch from 0x015126 to 0x01db2a (0x8a05 bytes) 000593 1786628940.008033 fallocate 0x030f22 thru 0x032bac (0x1c8b bytes) 000594 1786628940.014562 punch from 0x016764 to 0x02233d (0xbbda bytes) 000595 1786628940.019796 read 0x01b42e thru 0x02062d (0x5200 bytes) 000596 1786628940.030804 mapread 0x0253fa thru 0x034a8b (0xf692 bytes) 000597 1786628940.033493 mapread 0x038680 thru 0x03a1af (0x1b30 bytes) 000598 1786628940.034710 write 0x013f3c thru 0x01b598 (0x765d bytes) 000599 1786628940.038724 trunc from 0x03a1b0 to 0x0071fe (0x32fb1 bytes) 000600 1786628940.058641 mapread 0x00564f thru 0x0071fe (0x1bb0 bytes) 000601 1786628940.060873 mapwrite 0x0043cc thru 0x00dea5 (0x9ada bytes) 000602 1786628940.127596 mapread 0x00cf44 thru 0x00dea5 (0xf62 bytes) 000604 1786628940.130064 trunc from 0x0000a2 to 0x0037b4 (0x3713 bytes) 000605 1786628940.136956 write 0x033982 thru 0x035039 (0x16b8 bytes) 000606 1786628940.139016 trunc from 0x001d01 to 0x0052da (0x35da bytes) 000608 1786628940.146669 write 0x033af6 thru 0x036f9f (0x34aa bytes) 000610 1786628940.149948 punch from 0x00b8a6 to 0x01148e (0x5be9 bytes) 000611 1786628940.157467 write 0x034052 thru 0x03ffff (0xbfae bytes) 000612 1786628940.160612 write 0x014ffc thru 0x019269 (0x426e bytes) 000614 1786628940.170736 read 0x015a4e thru 0x01f999 (0x9f4c bytes) 000615 1786628940.179073 fallocate 0x027bd7 thru 0x02ea57 (0x6e81 bytes) 000616 1786628940.186839 read 0x019314 thru 0x020450 (0x713d bytes) 000617 1786628940.187573 write 0x00443b thru 0x00bf1b (0x7ae1 bytes) 000618 1786628940.189003 write 0x0205ae thru 0x029e9f (0x98f2 bytes) 000619 1786628940.191229 punch from 0x00d8a2 to 0x018f5d (0xb6bc bytes) 000620 1786628940.223158 fallocate 0x007d82 thru 0x00cb2f (0x4dae bytes) 000621 1786628940.233247 write 0x009373 thru 0x00b928 (0x25b6 bytes) 000622 1786628940.235996 mapread 0x02e7ec thru 0x033c23 (0x5438 bytes) 000623 1786628940.238034 write 0x035ddb thru 0x03eaca (0x8cf0 bytes) 000624 1786628940.246004 fallocate 0x035d72 thru 0x03c36f (0x65fe bytes) 000625 1786628940.266731 trunc from 0x040000 to 0x02976c (0x16893 bytes) 000626 1786628940.306402 read 0x014111 thru 0x0219f2 (0xd8e2 bytes) 000628 1786628940.323209 write 0x00376a thru 0x009b76 (0x640d bytes) 000629 1786628940.334792 mapread 0x00fdb2 thru 0x017052 (0x72a1 bytes) 000630 1786628940.336174 punch from 0x021566 to 0x02976c (0x8207 bytes) 000631 1786628940.351086 fallocate 0x01cc2d thru 0x0286ac (0xba80 bytes) 000632 1786628940.363663 trunc from 0x01d561 to 0x01f41d (0x1ebd bytes) 000633 1786628940.370910 read 0x025923 thru 0x02976c (0x3e4a bytes) 000634 1786628940.384852 read 0x01d889 thru 0x026c8d (0x9405 bytes) 000636 1786628940.388753 write 0x00ec72 thru 0x00f59a (0x929 bytes) 000637 1786628940.391634 write 0x02580e thru 0x02659a (0xd8d bytes) 000638 1786628940.393383 trunc from 0x003027 to 0x00e220 (0xb1fa bytes) 000639 1786628940.423671 trunc from 0x02976d to 0x025cb7 (0x3ab5 bytes) 000640 1786628940.446694 mapwrite 0x012221 thru 0x01da5e (0xb83e bytes) 000641 1786628940.468991 read 0x00fa72 thru 0x01e177 (0xe706 bytes) 000642 1786628940.470543 mapwrite 0x002032 thru 0x00f14b (0xd11a bytes) 000643 1786628940.503565 trunc from 0x025cb8 to 0x029cdc (0x4025 bytes) 000644 1786628940.527999 trunc from 0x029cdd to 0x00b9e9 (0x1e2f3 bytes) 000645 1786628940.544943 mapwrite 0x00acbf thru 0x013943 (0x8c85 bytes) 000647 1786628940.623133 punch from 0x00a11a to 0x011140 (0x7027 bytes) 000648 1786628940.636327 trunc from 0x013944 to 0x02c3af (0x18a6c bytes) 000650 1786628940.658846 mapread 0x0005d9 thru 0x002e48 (0x2870 bytes) 000651 1786628940.660563 write 0x01985c thru 0x024ab5 (0xb25a bytes) 000654 1786628940.695905 write 0x027e11 thru 0x030bc0 (0x8db0 bytes) 000655 1786628940.699029 fallocate 0x016cf0 thru 0x01b863 (0x4b74 bytes) 000658 1786628940.712557 mapwrite 0x01fd25 thru 0x025c20 (0x5efc bytes) 000659 1786628940.758266 trunc from 0x019ea1 to 0x026c42 (0xcda2 bytes) 000660 1786628940.776494 write 0x002642 thru 0x01202e (0xf9ed bytes) 000663 1786628940.787794 read 0x00ebcf thru 0x017c17 (0x9049 bytes) 000664 1786628940.809591 trunc from 0x000f63 to 0x005ada (0x4b78 bytes) 000665 1786628940.829651 mapread 0x02845b thru 0x030bc0 (0x8766 bytes) 000666 1786628940.830973 trunc from 0x030bc1 to 0x005662 (0x2b55e bytes) 000667 1786628940.866706 fallocate 0x03772a thru 0x03ffff (0x88d6 bytes) 000668 1786628940.876841 read 0x003031 thru 0x0031c0 (0x190 bytes) 000669 1786628940.893201 mapwrite 0x03a7d5 thru 0x03ffff (0x582b bytes) 000670 1786628940.942487 mapwrite 0x0246c1 thru 0x031278 (0xcbb8 bytes) 000673 1786628940.994981 fallocate 0x023943 thru 0x02e488 (0xab46 bytes) 000674 1786628941.003766 punch from 0x0021be to 0x00af12 (0x8d55 bytes) 000675 1786628941.019223 trunc from 0x01de7e to 0x0264e0 (0x8663 bytes) 000677 1786628941.030009 trunc from 0x01a561 to 0x01f24a (0x4cea bytes) 000678 1786628941.041629 trunc from 0x040000 to 0x02e349 (0x11cb6 bytes) 000679 1786628941.064695 fallocate 0x0257ba thru 0x02a42f (0x4c76 bytes) 000680 1786628941.071620 mapwrite 0x02515a thru 0x02b375 (0x621c bytes) 000681 1786628941.104372 write 0x032f4c thru 0x03ffff (0xd0b4 bytes) 000682 1786628941.107673 trunc from 0x034162 to 0x03bcd2 (0x7b71 bytes) 000683 1786628941.130271 write 0x0182e1 thru 0x01f06d (0x6d8d bytes) 000685 1786628941.137863 trunc from 0x040000 to 0x00a493 (0x35b6c bytes) 000686 1786628941.159425 trunc from 0x00a494 to 0x005217 (0x527c bytes) 000687 1786628941.179154 mapwrite 0x013a1e thru 0x019bb7 (0x619a bytes) 000688 1786628941.275970 punch from 0x005ea3 to 0x0156e0 (0xf83e bytes) 000689 1786628941.284215 trunc from 0x00255d to 0x005b05 (0x35a9 bytes) 000690 1786628941.293720 read 0x00ce5b thru 0x0198e9 (0xca8f bytes) 000691 1786628941.313724 trunc from 0x017be0 to 0x019bb7 (0x1fd8 bytes) 000692 1786628941.320833 read 0x00f6da thru 0x00fa8d (0x3b4 bytes) 000694 1786628941.325892 fallocate 0x03025c thru 0x03b90f (0xb6b4 bytes) 000695 1786628941.334508 write 0x0262aa thru 0x031f17 (0xbc6e bytes) 000696 1786628941.338913 fallocate 0x03ea99 thru 0x03f3b7 (0x91f bytes) 000697 1786628941.347641 mapread 0x03a31f thru 0x03f3b7 (0x5099 bytes) 000699 1786628941.375074 mapread 0x013e69 thru 0x01d79b (0x9933 bytes) 000700 1786628941.377921 mapwrite 0x017288 thru 0x021c32 (0xa9ab bytes) 000701 1786628941.406863 fallocate 0x01ce53 thru 0x0232c9 (0x6477 bytes) 000702 1786628941.413834 punch from 0x036020 to 0x03c42a (0x640b bytes) 000703 1786628941.420933 trunc from 0x0365f7 to 0x0380b7 (0x1ac1 bytes) 000704 1786628941.427009 read 0x037859 thru 0x03f3b7 (0x7b5f bytes) 000706 1786628941.445523 MIRROR_EXTEND to 2 mirrors 000707 1786628942.745968 MIRROR_RESYNC 2 mirrors 000708 1786628943.348669 fallocate 0x012535 thru 0x01e361 (0xbe2d bytes) 000709 1786628943.421782 fallocate 0x03feaf thru 0x03ffff (0x151 bytes) 000710 1786628943.429374 punch from 0x007c5a to 0x01483b (0xcbe2 bytes) 000711 1786628943.444024 read 0x002f16 thru 0x00b56f (0x865a bytes) 000712 1786628943.460365 trunc from 0x040000 to 0x035e90 (0xa16f bytes) 000714 1786628943.531582 punch from 0x0079f4 to 0x0160a2 (0xe6af bytes) 000715 1786628943.549233 mapwrite 0x01f316 thru 0x023c29 (0x4914 bytes) 000716 1786628943.653474 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35e91): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e91): Mirror 2: 0x3236c927 (stale) CRC-32 checksum value for chunk [0, 0x35e91): Mirror 1: 0xba6e1775 Mirror 3: 0xba6e1775 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e91: skipped 000717 1786628944.988224 trunc from 0x035e91 to 0x02306d (0x12e23 bytes) 000718 1786628945.106759 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2306e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2306e): Mirror 2: 0x6fd601a0 (stale) Mirror 3: 0x3319c689 (stale) CRC-32 checksum value for chunk [0, 0x2306e): Mirror 1: 0x3319c689 Mirror 4: 0x3319c689 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2306e: skipped 000719 1786628946.143544 write 0x020281 thru 0x02e17f (0xdeff bytes) 000720 1786628946.185115 write 0x02b902 thru 0x02bbb3 (0x2b2 bytes) 000721 1786628946.185858 mapwrite 0x0338cc thru 0x03f5e1 (0xbd16 bytes) 000722 1786628946.253441 mapwrite 0x03f2fb thru 0x03f6f4 (0x3fa bytes) 000723 1786628946.294411 mapread 0x003155 thru 0x00ddac (0xac58 bytes) 000724 1786628946.311142 MIRROR_RESYNC 4 mirrors 000725 1786628947.390469 mapwrite 0x029e1e thru 0x031a4b (0x7c2e bytes) 000726 1786628947.503686 mapwrite 0x00a232 thru 0x011685 (0x7454 bytes) 000727 1786628947.529473 trunc from 0x02cd8b to 0x034a6d (0x7ce3 bytes) 000728 1786628947.534579 mapread 0x0163bf thru 0x0204d2 (0xa114 bytes) 000729 1786628947.553922 punch from 0x013260 to 0x01394f (0x6f0 bytes) 000730 1786628947.562480 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f6f5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6f5): Mirror 2: 0x2562361f (stale) Mirror 3: 0x2562361f (stale) Mirror 4: 0x2562361f (stale) CRC-32 checksum value for chunk [0, 0x3f6f5): Mirror 1: 0x759b1d33 Mirror 5: 0x759b1d33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6f5: skipped 000731 1786628948.674673 trunc from 0x03f6f5 to 0x02863a (0x170ba bytes) 000732 1786628948.744513 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2863b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2863b): Mirror 2: 0xf3fbbac7 (stale) Mirror 3: 0xf3fbbac7 (stale) Mirror 4: 0xf3fbbac7 (stale) Mirror 5: 0xdb0d24a3 (stale) CRC-32 checksum value for chunk [0, 0x2863b): Mirror 1: 0xdb0d24a3 Mirror 6: 0xdb0d24a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2863b: skipped 000733 1786628949.832133 punch from 0x00cce4 to 0x01a3b0 (0xd6cd bytes) 000734 1786628949.889996 mapwrite 0x02a682 thru 0x0352e7 (0xac66 bytes) 000735 1786628949.997975 read 0x00eaf4 thru 0x01b42b (0xc938 bytes) 000736 1786628950.016871 read 0x000c71 thru 0x00273a (0x1aca bytes) 000737 1786628950.031116 trunc from 0x022267 to 0x02af0f (0x8ca9 bytes) 000738 1786628950.036875 mapwrite 0x035187 thru 0x0377f4 (0x266e bytes) 000739 1786628950.080624 mapwrite 0x01879f thru 0x0250ca (0xc92c bytes) 000740 1786628950.142682 mapread 0x00b542 thru 0x010887 (0x5346 bytes) 000741 1786628950.144322 mapread 0x007bc1 thru 0x00c6c9 (0x4b09 bytes) 000742 1786628950.146369 trunc from 0x00f71c to 0x01d5d6 (0xdebb bytes) 000743 1786628950.156252 fallocate 0x025d2e thru 0x028d23 (0x2ff6 bytes) 000744 1786628950.168912 mapwrite 0x03547a thru 0x03c01f (0x6ba6 bytes) 000745 1786628950.253058 mapwrite 0x002edf thru 0x00e39e (0xb4c0 bytes) 000746 1786628950.294433 punch from 0x018305 to 0x018950 (0x64c bytes) 000749 1786628950.304572 fallocate 0x026ab3 thru 0x030a61 (0x9faf bytes) 000750 1786628950.309402 read 0x0320fa thru 0x03c01f (0x9f26 bytes) 000751 1786628950.310825 punch from 0x007316 to 0x01046f (0x915a bytes) 000752 1786628950.318433 trunc from 0x01bd58 to 0x01d733 (0x19dc bytes) 000753 1786628950.326130 mapwrite 0x00e280 thru 0x013b5f (0x58e0 bytes) 000754 1786628950.367637 trunc from 0x01bd13 to 0x026c07 (0xaef5 bytes) 000755 1786628950.375848 mapwrite 0x022442 thru 0x028647 (0x6206 bytes) 000756 1786628950.437838 punch from 0x01c069 to 0x02007c (0x4014 bytes) 000757 1786628950.444948 read 0x00c9b8 thru 0x014627 (0x7c70 bytes) 000758 1786628950.448907 fallocate 0x00e780 thru 0x00f8a9 (0x112a bytes) 000759 1786628950.456056 punch from 0x00de4b to 0x0115f1 (0x37a7 bytes) 000760 1786628950.463081 write 0x003a59 thru 0x00fe8e (0xc436 bytes) 000761 1786628950.470665 mapread 0x0277d2 thru 0x036176 (0xe9a5 bytes) 000763 1786628950.473611 write 0x034c5c thru 0x03f779 (0xab1e bytes) 000764 1786628950.475183 mapwrite 0x031581 thru 0x036fb1 (0x5a31 bytes) 000765 1786628950.495832 fallocate 0x00c753 thru 0x00fdec (0x369a bytes) 000766 1786628950.501756 write 0x03e9ed thru 0x03ffff (0x1613 bytes) 000767 1786628950.506640 trunc from 0x0005bf to 0x00ce20 (0xc862 bytes) 000768 1786628950.513935 mapwrite 0x003108 thru 0x0127cf (0xf6c8 bytes) 000769 1786628950.566958 punch from 0x00ec88 to 0x01e56e (0xf8e7 bytes) 000770 1786628950.577810 mapread 0x01eab5 thru 0x02dc9c (0xf1e8 bytes) 000771 1786628950.605184 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x69f4fda2 (stale) Mirror 3: 0x69f4fda2 (stale) Mirror 5: 0xe7aa070d (stale) Mirror 6: 0xeb48d492 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x69f4fda2 (stale) Mirror 3: 0x69f4fda2 (stale) Mirror 5: 0xe7aa070d (stale) Mirror 6: 0xeb48d492 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000772 1786628951.096721 punch from 0x03c5f7 to 0x03ffff (0x3a09 bytes) 000773 1786628951.106187 fallocate 0x027def thru 0x02d1e3 (0x53f5 bytes) 000774 1786628951.111592 mapread 0x039a1b thru 0x03ffff (0x65e5 bytes) 000775 1786628951.128592 mapwrite 0x017ef9 thru 0x023628 (0xb730 bytes) 000776 1786628951.184641 write 0x01c0d7 thru 0x024127 (0x8051 bytes) 000777 1786628951.201036 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x69f4fda2 (stale) Mirror 3: 0x69f4fda2 (stale) Mirror 5: 0xe7aa070d (stale) Mirror 6: 0xeb48d492 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x35b402f1 Mirror 7: 0x35b402f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000778 1786628952.305943 mapwrite 0x00f786 thru 0x013b1a (0x4395 bytes) 000779 1786628952.461248 read 0x038d37 thru 0x03c04b (0x3315 bytes) 000780 1786628952.476786 mapwrite 0x02bedb thru 0x02bf02 (0x28 bytes) 000781 1786628952.500813 punch from 0x00fd63 to 0x01f507 (0xf7a5 bytes) 000782 1786628952.507720 read 0x018ec6 thru 0x01fdc7 (0x6f02 bytes) 000783 1786628952.520869 trunc from 0x03de4c to 0x03ffff (0x21b4 bytes) 000784 1786628952.531597 write 0x037a45 thru 0x03ffff (0x85bb bytes) 000785 1786628952.544890 trunc from 0x015cd8 to 0x01f5f3 (0x991c bytes) 000786 1786628952.560449 trunc from 0x032a02 to 0x03ffff (0xd5fe bytes) 000787 1786628952.584866 read 0x035008 thru 0x03d36f (0x8368 bytes) 000788 1786628952.595910 trunc from 0x012172 to 0x01862c (0x64bb bytes) 000789 1786628952.601666 mapread 0x01436a thru 0x022d44 (0xe9db bytes) 000790 1786628952.629776 punch from 0x006ac3 to 0x010645 (0x9b83 bytes) 000791 1786628952.637928 mapwrite 0x00923a thru 0x00fed2 (0x6c99 bytes) 000792 1786628952.700807 fallocate 0x0248de thru 0x029bf2 (0x5315 bytes) 000793 1786628952.711713 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x69f4fda2 (stale) Mirror 3: 0x69f4fda2 (stale) Mirror 5: 0xe7aa070d (stale) Mirror 7: 0x35b402f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x69f4fda2 (stale) Mirror 3: 0x69f4fda2 (stale) Mirror 5: 0xe7aa070d (stale) Mirror 7: 0x35b402f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000794 1786628953.309654 trunc from 0x00e063 to 0x011e42 (0x3de0 bytes) 000795 1786628953.329812 mapread 0x008744 thru 0x00d586 (0x4e43 bytes) 000796 1786628953.368637 trunc from 0x040000 to 0x0115b2 (0x2ea4d bytes) 000797 1786628953.473206 write 0x03f5c8 thru 0x03ffff (0xa38 bytes) 000798 1786628953.476687 trunc from 0x01d4c3 to 0x022d60 (0x589e bytes) 000799 1786628953.484952 read 0x022a4f thru 0x0258e3 (0x2e95 bytes) 000800 1786628953.500589 trunc from 0x037d0a to 0x03b1a2 (0x3499 bytes) 000801 1786628953.513141 read 0x02a176 thru 0x02afbc (0xe47 bytes) 000802 1786628953.532515 punch from 0x00652a to 0x00dddd (0x78b4 bytes) 000803 1786628953.539448 trunc from 0x02d800 to 0x03b53c (0xdd3d bytes) 000804 1786628953.547351 write 0x00d830 thru 0x013281 (0x5a52 bytes) 000805 1786628953.560940 punch from 0x005539 to 0x0126c5 (0xd18d bytes) 000806 1786628953.595209 punch from 0x03dc80 to 0x03ffff (0x2380 bytes) 000808 1786628953.611218 read 0x036f5e thru 0x03e577 (0x761a bytes) 000809 1786628953.630722 mapread 0x028f31 thru 0x029a14 (0xae4 bytes) 000810 1786628953.631971 trunc from 0x02cd8e to 0x039e4a (0xd0bd bytes) 000811 1786628953.648503 mapwrite 0x035f35 thru 0x03914d (0x3219 bytes) 000812 1786628953.717662 mapwrite 0x01edc9 thru 0x0209c0 (0x1bf8 bytes) 000813 1786628953.747627 write 0x001a46 thru 0x009937 (0x7ef2 bytes) 000814 1786628953.748925 fallocate 0x02e39c thru 0x03d159 (0xedbe bytes) 000815 1786628953.764900 mapwrite 0x037cce thru 0x038ab1 (0xde4 bytes) 000816 1786628953.805406 fallocate 0x039ece thru 0x03e300 (0x4433 bytes) 000817 1786628953.810578 punch from 0x01fd3f to 0x026aca (0x6d8c bytes) 000818 1786628953.816295 mapread 0x002cbe thru 0x006c50 (0x3f93 bytes) 000819 1786628953.818507 punch from 0x00bf45 to 0x018161 (0xc21d bytes) 000820 1786628953.823174 fallocate 0x00a647 thru 0x00c8ff (0x22b9 bytes) 000821 1786628953.828505 read 0x034e1f thru 0x03f194 (0xa376 bytes) 000822 1786628953.854600 mapwrite 0x00c04c thru 0x00e155 (0x210a bytes) 000823 1786628953.886621 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x69f4fda2 (stale) Mirror 5: 0xe7aa070d (stale) Mirror 7: 0x35b402f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x69f4fda2 (stale) Mirror 5: 0xe7aa070d (stale) Mirror 7: 0x35b402f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000824 1786628954.464395 MIRROR_RESYNC 4 mirrors 000826 1786628956.261582 trunc from 0x01b6d6 to 0x0273ff (0xbd2a bytes) 000827 1786628956.298599 read 0x03fb2a thru 0x03ffff (0x4d6 bytes) 000830 1786628956.311486 write 0x00010d thru 0x008357 (0x824b bytes) 000831 1786628956.333334 trunc from 0x034961 to 0x0356c2 (0xd62 bytes) 000832 1786628956.342492 read 0x004f5c thru 0x007864 (0x2909 bytes) 000834 1786628956.344611 fallocate 0x010709 thru 0x012aaf (0x23a7 bytes) 000836 1786628956.372655 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbadd787f (stale) Mirror 5: 0xbadd787f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xbadd787f (stale) Mirror 5: 0xbadd787f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000838 1786628956.918733 fallocate 0x03e790 thru 0x03edbc (0x62d bytes) 000839 1786628956.930114 trunc from 0x040000 to 0x00c51b (0x33ae4 bytes) 000840 1786628956.968020 punch from 0x006061 to 0x00c487 (0x6427 bytes) 000841 1786628956.979402 punch from 0x002706 to 0x008824 (0x611f bytes) 000842 1786628956.987343 write 0x0296c2 thru 0x02adf6 (0x1735 bytes) 000843 1786628956.989498 mapread 0x013461 thru 0x01796a (0x450a bytes) 000844 1786628957.025985 mapread 0x014276 thru 0x023f52 (0xfcdd bytes) 000845 1786628957.068662 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2adf7): Mirror 5: 0x6d470f18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2adf7): Mirror 5: 0x6d470f18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2adf7: skipped 000846 1786628957.573949 read 0x020c7a thru 0x02124f (0x5d6 bytes) 000847 1786628957.586167 fallocate 0x032cb5 thru 0x032fe7 (0x333 bytes) 000848 1786628957.605537 read 0x006c0d thru 0x00e282 (0x7676 bytes) 000849 1786628957.631787 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2adf7): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2adf7): Mirror 5: 0x6d470f18 (stale) CRC-32 checksum value for chunk [0, 0x2adf7): Mirror 1: 0xf320c408 Mirror 6: 0xf320c408 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2adf7: skipped 000850 1786628958.810516 mapwrite 0x009a8f thru 0x015e72 (0xc3e4 bytes) 000851 1786628958.953237 fallocate 0x000477 thru 0x00684a (0x63d4 bytes) 000852 1786628958.960234 punch from 0x005a45 to 0x00a69d (0x4c59 bytes) 000853 1786628958.974748 punch from 0x01ab53 to 0x02a39a (0xf848 bytes) 000854 1786628958.984762 mapread 0x00e1aa thru 0x011873 (0x36ca bytes) 000855 1786628958.986123 mapread 0x0042b9 thru 0x00511c (0xe64 bytes) 000856 1786628959.003811 trunc from 0x013b7f to 0x01da2e (0x9eb0 bytes) 000857 1786628959.012936 trunc from 0x015df3 to 0x01e8bc (0x8aca bytes) 000858 1786628959.021418 punch from 0x019a15 to 0x01e64e (0x4c3a bytes) 000859 1786628959.027259 mapread 0x011b59 thru 0x0180bb (0x6563 bytes) 000860 1786628959.058715 fallocate 0x013fd0 thru 0x0222df (0xe310 bytes) 000861 1786628959.064029 trunc from 0x00fe98 to 0x019442 (0x95ab bytes) 000862 1786628959.071614 trunc from 0x00747e to 0x0163b0 (0xef33 bytes) 000864 1786628959.079917 fallocate 0x03df2f thru 0x03ffff (0x20d1 bytes) 000866 1786628959.088999 mapwrite 0x034256 thru 0x035446 (0x11f1 bytes) 000867 1786628959.197829 trunc from 0x00b3f3 to 0x01043c (0x504a bytes) 000869 1786628959.215607 MIRROR_RESYNC 3 mirrors 000870 1786628960.248598 mapread 0x01e3ad thru 0x023f37 (0x5b8b bytes) 000871 1786628960.286367 write 0x00f2c2 thru 0x012844 (0x3583 bytes) 000872 1786628960.348089 read 0x015176 thru 0x023b88 (0xea13 bytes) 000873 1786628960.365018 read 0x000e58 thru 0x00c6a6 (0xb84f bytes) 000874 1786628960.380248 trunc from 0x035447 to 0x02a2c2 (0xb184 bytes) 000875 1786628960.434330 fallocate 0x03d080 thru 0x03ffff (0x2f80 bytes) 000876 1786628960.451825 fallocate 0x00e16d thru 0x010603 (0x2497 bytes) 000877 1786628960.468526 punch from 0x019f49 to 0x01bdd6 (0x1e8e bytes) 000878 1786628960.473734 fallocate 0x0062ae thru 0x010d8a (0xaadd bytes) 000879 1786628960.481088 mapwrite 0x03ac4c thru 0x03ffff (0x53b4 bytes) 000880 1786628960.534534 fallocate 0x01a470 thru 0x027939 (0xd4ca bytes) 000881 1786628960.540256 trunc from 0x040000 to 0x0028bb (0x3d744 bytes) 000882 1786628960.587990 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28bc): Mirror 5: 0xd0046230 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28bc): Mirror 5: 0xd0046230 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28bc: skipped 000883 1786628960.991700 mapread 0x000149 thru 0x0028bb (0x2773 bytes) 000885 1786628961.023835 read 0x000236 thru 0x0028bb (0x2686 bytes) 000886 1786628961.025763 punch from 0x001f5a to 0x0028bb (0x962 bytes) 000887 1786628961.040059 punch from 0x0002b7 to 0x0020f0 (0x1e3a bytes) 000888 1786628961.049026 mapread 0x001964 thru 0x0028bb (0xf58 bytes) 000889 1786628961.067661 write 0x01bfc9 thru 0x0204a5 (0x44dd bytes) 000890 1786628961.079861 trunc from 0x0204a6 to 0x0213ec (0xf47 bytes) 000891 1786628961.101063 fallocate 0x03a969 thru 0x03d559 (0x2bf1 bytes) 000892 1786628961.111018 read 0x00fea7 thru 0x01376e (0x38c8 bytes) 000893 1786628961.126010 fallocate 0x01b43d thru 0x026cc2 (0xb886 bytes) 000894 1786628961.143073 punch from 0x027086 to 0x02be1b (0x4d96 bytes) 000895 1786628961.149002 write 0x035e7d thru 0x03f8fa (0x9a7e bytes) 000896 1786628961.157096 mapread 0x036072 thru 0x03e3e9 (0x8378 bytes) 000897 1786628961.166836 mapread 0x01e3aa thru 0x02c27f (0xded6 bytes) 000898 1786628961.241691 punch from 0x02bcd1 to 0x035db9 (0xa0e9 bytes) 000899 1786628961.280501 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f8fb): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8fb): Mirror 5: 0xcf59468e (stale) CRC-32 checksum value for chunk [0, 0x3f8fb): Mirror 1: 0xa05c6968 Mirror 6: 0xa05c6968 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8fb: skipped 000900 1786628962.378465 trunc from 0x03f8fb to 0x000f9f (0x3e95b bytes) 000901 1786628962.461362 trunc from 0x000fa0 to 0x026bb2 (0x25c13 bytes) 000902 1786628962.494088 mapread 0x014716 thru 0x0184a2 (0x3d8d bytes) 000903 1786628962.521277 trunc from 0x026bb3 to 0x021ecd (0x4ce5 bytes) 000904 1786628962.541075 read 0x015066 thru 0x01bea3 (0x6e3e bytes) 000905 1786628962.542090 punch from 0x01e7df to 0x021ecd (0x36ef bytes) 000906 1786628962.547677 read 0x003d53 thru 0x00b38c (0x763a bytes) 000908 1786628962.557832 fallocate 0x02d45c thru 0x038694 (0xb239 bytes) 000909 1786628962.563680 mapwrite 0x022228 thru 0x023a68 (0x1841 bytes) 000910 1786628962.595632 punch from 0x0233b1 to 0x023a68 (0x6b8 bytes) 000911 1786628962.602100 write 0x031a5f thru 0x0380ee (0x6690 bytes) 000912 1786628962.604699 mapwrite 0x036792 thru 0x0393de (0x2c4d bytes) 000915 1786628962.660672 fallocate 0x0067e6 thru 0x0148a9 (0xe0c4 bytes) 000916 1786628962.667022 mapread 0x02b3d6 thru 0x0393de (0xe009 bytes) 000917 1786628962.669791 mapwrite 0x0227ad thru 0x02390b (0x115f bytes) 000918 1786628962.701581 fallocate 0x039279 thru 0x039b75 (0x8fd bytes) 000919 1786628962.715766 mapwrite 0x00f4b8 thru 0x014911 (0x545a bytes) 000920 1786628962.747078 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x393df): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x393df): Mirror 5: 0xc3f87629 (stale) Mirror 6: 0x1daf66d2 (stale) CRC-32 checksum value for chunk [0, 0x393df): Mirror 1: 0xcf6077e9 Mirror 7: 0xcf6077e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x393df: skipped 000921 1786628963.846996 fallocate 0x001f0f thru 0x004d23 (0x2e15 bytes) 000922 1786628963.887353 trunc from 0x0393df to 0x005d24 (0x336ba bytes) 000923 1786628963.920605 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x5d25): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5d25): Mirror 5: 0x2730d37e (stale) Mirror 6: 0xafb9aee1 (stale) Mirror 7: 0xafb9aee1 (stale) CRC-32 checksum value for chunk [0, 0x5d25): Mirror 1: 0xafb9aee1 Mirror 8: 0xafb9aee1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5d25: skipped 000924 1786628964.895056 mapwrite 0x025a7a thru 0x031b3e (0xc0c5 bytes) 000925 1786628965.039713 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31b3f): Mirror 5: 0x7ad198e1 (stale) Mirror 6: 0xfaa7953a (stale) Mirror 7: 0x6ead08ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b3f): Mirror 5: 0x7ad198e1 (stale) Mirror 6: 0xfaa7953a (stale) Mirror 7: 0x6ead08ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b3f: skipped 000926 1786628965.490542 punch from 0x031a2b to 0x031b3e (0x114 bytes) 000930 1786628965.502909 trunc from 0x031b3f to 0x01f962 (0x121dc bytes) 000931 1786628965.533238 write 0x01b6df thru 0x029327 (0xdc49 bytes) 000932 1786628965.540686 read 0x012d61 thru 0x01fd50 (0xcff0 bytes) 000933 1786628965.550942 read 0x012184 thru 0x01635e (0x41db bytes) 000935 1786628965.552393 mapwrite 0x037820 thru 0x03addb (0x35bc bytes) 000936 1786628965.644062 fallocate 0x02d9de thru 0x03228e (0x48b1 bytes) 000937 1786628965.654914 fallocate 0x03f375 thru 0x03ffff (0xc8b bytes) 000939 1786628965.671959 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3addc): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3addc): Mirror 5: 0xa5a7f356 (stale) Mirror 6: 0xfbfdb7bb (stale) Mirror 7: 0xd3f87283 (stale) CRC-32 checksum value for chunk [0, 0x3addc): Mirror 1: 0x28bfaffa Mirror 8: 0x28bfaffa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3addc: skipped 000940 1786628966.769079 write 0x037f31 thru 0x03ffff (0x80cf bytes) 000941 1786628966.846171 trunc from 0x040000 to 0x018d80 (0x2727f bytes) 000942 1786628966.872540 write 0x01c2cb thru 0x01fd04 (0x3a3a bytes) 000943 1786628966.884286 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fd05): Mirror 6: 0xd59f4be6 (stale) Mirror 7: 0xf36729c8 (stale) Mirror 8: 0x7f67e02b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fd05): Mirror 6: 0xd59f4be6 (stale) Mirror 7: 0xf36729c8 (stale) Mirror 8: 0x7f67e02b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fd05: skipped 000944 1786628967.307568 write 0x02746a thru 0x02f30b (0x7ea2 bytes) 000945 1786628967.320923 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f30c): Mirror 7: 0x66e720b9 (stale) Mirror 8: 0x3b744483 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f30c): Mirror 7: 0x66e720b9 (stale) Mirror 8: 0x3b744483 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f30c: skipped 000946 1786628967.755506 trunc from 0x02f30c to 0x014422 (0x1aee9 bytes) 000947 1786628967.797971 fallocate 0x035e46 thru 0x03bcab (0x5e66 bytes) 000948 1786628967.808745 punch from 0x00269b to 0x011d3d (0xf6a3 bytes) 000949 1786628967.821068 mapread 0x027b2c thru 0x03543c (0xd911 bytes) 000950 1786628967.856543 trunc from 0x001c06 to 0x00ff62 (0xe35d bytes) 000951 1786628967.862240 mapread 0x01ebd4 thru 0x0250d9 (0x6506 bytes) 000954 1786628967.877653 trunc from 0x022244 to 0x02b095 (0x8e52 bytes) 000955 1786628967.882875 punch from 0x0390f7 to 0x03bcab (0x2bb5 bytes) 000956 1786628967.888506 mapread 0x01bde5 thru 0x01e2bd (0x24d9 bytes) 000957 1786628967.890129 mapread 0x03b9cc thru 0x03bcab (0x2e0 bytes) 000958 1786628967.896510 mapread 0x01f281 thru 0x023603 (0x4383 bytes) 000959 1786628967.913266 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bcac): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bcac): Mirror 7: 0xd36c7332 (stale) Mirror 8: 0x8eba8217 (stale) CRC-32 checksum value for chunk [0, 0x3bcac): Mirror 1: 0xf871d0ff Mirror 9: 0xf871d0ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bcac: skipped 000960 1786628968.904959 mapread 0x02a982 thru 0x037050 (0xc6cf bytes) 000961 1786628968.951611 punch from 0x03a124 to 0x03bcab (0x1b88 bytes) 000962 1786628969.000783 punch from 0x012016 to 0x01a40d (0x83f8 bytes) 000963 1786628969.008620 fallocate 0x0210fa thru 0x0248ad (0x37b4 bytes) 000964 1786628969.016244 trunc from 0x007204 to 0x007848 (0x645 bytes) 000965 1786628969.022559 punch from 0x02e503 to 0x035409 (0x6f07 bytes) 000966 1786628969.042766 mapread 0x0228d2 thru 0x02c355 (0x9a84 bytes) 000967 1786628969.081170 trunc from 0x03bcac to 0x024d21 (0x16f8a bytes) 000968 1786628969.119460 punch from 0x00a7d4 to 0x017db4 (0xd5e1 bytes) 000969 1786628969.126593 punch from 0x00ce98 to 0x0186a0 (0xb809 bytes) 000970 1786628969.132211 read 0x02331e thru 0x024d21 (0x1a04 bytes) 000971 1786628969.133103 read 0x00245a thru 0x009460 (0x7007 bytes) 000972 1786628969.154795 write 0x02a5dc thru 0x034bfc (0xa621 bytes) 000973 1786628969.158650 write 0x03076b thru 0x039e21 (0x96b7 bytes) 000975 1786628969.160910 trunc from 0x039e22 to 0x031bd0 (0x8251 bytes) 000976 1786628969.181212 mapread 0x02551c thru 0x02971d (0x4202 bytes) 000978 1786628969.193680 punch from 0x02e1cc to 0x031bd0 (0x3a05 bytes) 000979 1786628969.210683 trunc from 0x026d55 to 0x0299b7 (0x2c63 bytes) 000980 1786628969.215555 punch from 0x01ee24 to 0x02a7fa (0xb9d7 bytes) 000981 1786628969.221928 write 0x03efbd thru 0x03ffff (0x1043 bytes) 000982 1786628969.226022 read 0x006268 thru 0x00d1ab (0x6f44 bytes) 000984 1786628969.228266 read 0x02c35e thru 0x03bb36 (0xf7d9 bytes) 000985 1786628969.246468 fallocate 0x013e32 thru 0x020919 (0xcae8 bytes) 000986 1786628969.251712 mapwrite 0x02eea8 thru 0x03461f (0x5778 bytes) 000987 1786628969.306767 write 0x03bd23 thru 0x03ffff (0x42dd bytes) 000988 1786628969.311061 punch from 0x01f2e6 to 0x02f173 (0xfe8e bytes) 000989 1786628969.328268 fallocate 0x036b97 thru 0x03a426 (0x3890 bytes) 000990 1786628969.342880 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1c3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xafd02ce4 (stale) Mirror 8: 0xe1b7c4fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xafd02ce4 (stale) Mirror 8: 0xe1b7c4fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000991 1786628969.750042 fallocate 0x033dbb thru 0x03ffff (0xc245 bytes) 000992 1786628969.755632 fallocate 0x03bb9e thru 0x03f89b (0x3cfe bytes) 000994 1786628969.761193 trunc from 0x035578 to 0x03e19f (0x8c28 bytes) 000995 1786628969.767670 write 0x008027 thru 0x017dd2 (0xfdac bytes) 000996 1786628969.784027 trunc from 0x009f0e to 0x00c175 (0x2268 bytes) 000997 1786628969.821849 fallocate 0x0341ff thru 0x03c6d1 (0x84d3 bytes) 000998 1786628969.839555 mapwrite 0x030726 thru 0x03e903 (0xe1de bytes) 000999 1786628969.925285 trunc from 0x040000 to 0x0073be (0x38c41 bytes) 001000 1786628969.957080 MIRROR_RESYNC 3 mirrors All operations completed A-OK! PASS 70a (98s) == sanity-flr test 71: check mirror extend parent fid ==== 09:49:36 (1786628976) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x20a:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 09:49:45 (1786628985) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.177906 s, 11.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161507 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xeb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xec:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 09:49:53 (1786628993) Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2044064 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7836128 .. Extending file size to 7402080 .. done Extending file size to 7838688 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9299104 .. Extending file size to 7946240 .. Extending file size to 9236704 .. Extending file size to 9276768 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61379 61380 61382 61383 61386 done 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) 192.168.203.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:51:13 (1786629073) Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5052352 .. Extending file size to 4367936 .. Extending file size to 8184224 .. Extending file size to 5591168 .. Extending file size to 8372064 .. Extending file size to 8617632 .. Extending file size to 7046688 .. Extending file size to 7050720 .. Extending file size to 8525184 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8645664 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8949216 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9022432 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9061920 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9102080 .. Extending file size to 9216096 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63119 63120 63121 63124 63125 done 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) 192.168.203.117@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:52:29 (1786629149) Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2192 1269112 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25908 3581112 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17888 3589132 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43796 7170244 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2192 1269112 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25908 3581112 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17888 3589132 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43796 7170244 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115949 s, 18.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194839 s, 10.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xac60af5c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:52:49 (1786629169) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.157494 s, 13.3 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 09:52:57 (1786629177) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x3a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x123:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x123:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x101:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:53:03 (1786629183) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0933245 s, 22.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x256:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x256:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x257:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:53:10 (1786629190) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19374 s, 10.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x102:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x125:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x126:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x102:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x103:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:53:20 (1786629200) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13945 s, 15.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x104:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x104:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x105:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:53:56 (1786629236) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg317-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776665 s, 13.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x257:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x259:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x258:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x257:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x259:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x258:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x258:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:54:47 (1786629287) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg317-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x106:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x107:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0953678 s, 22.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x106:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x107:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x106:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x107:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x128:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:55:34 (1786629334) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg317-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg317-server: Pool lustre.test_204f created Adding targets to pool oleg317-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg317-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0950776 s, 22.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x25a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x259:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x25a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x259:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg317-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg317-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg317-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:56:49 (1786629409) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:56:57 (1786629417) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:57:05 (1786629425) oleg317-server: Pool lustre.test_206 created Adding targets to pool oleg317-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg317-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg317-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg317-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg317-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:57:31 (1786629451) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772807 s, 13.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254971 s, 41.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x57df932e Mirror 3: 0x57df932e PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:57:41 (1786629461) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:57:44 (1786629464) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:57:47 (1786629467) Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0640537 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x7d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x19b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x19b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x19b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x24d0000 Read this (4 bytes): BBBB 192.168.203.117@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg317-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:58:22 (1786629502) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0167844 s, 488 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:58:28 (1786629508) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601295 s, 17.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59 lmm_fid: [0x200000405:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x59 lmm_fid: [0x200000405:0x59:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611225 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-13 09:58:29.000000000 -0400 Modify: 2026-08-13 09:58:29.000000000 -0400 Change: 2026-08-13 09:58:31.000000000 -0400 Birth: 2026-08-13 09:58:30.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:58:36 (1786629516) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473429 s, 22.1 MB/s oleg317-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg317-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg317-server: O/280000401/d19/113: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:58:55 (1786629535) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.233131 s, 176 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0444952 s, 92.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 13 09:58 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 13 09:58 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:59:03 (1786629543) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:59:10 (1786629550) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1706 sec =========== 09:59:19 (1786629559) === sanity-flr: start cleanup 09:59:20 (1786629560) === === sanity-flr: finish cleanup 09:59:29 (1786629569) ===