-----============= acceptance-small: sanity-flr ============----- Fri Aug 14 21:14:53 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg356-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 21:15:13 (1786756513) === oleg356-client.virtnet: executing check_config_client /mnt/lustre oleg356-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg356-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e29e18b6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e29e18b6800.idle_timeout=debug disable quota as required oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 21:15:49 (1786756549) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8243] == sanity-flr test 0a: lfs mirror create with -N option == 21:15:51 (1786756551) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:16:10 (1786756570) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:16:13 (1786756573) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:16:16 (1786756576) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:16:39 (1786756599) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:16:41 (1786756601) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 21:16:45 (1786756605) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8e29e18b6800.stats=clear osc.lustre-OST0001-osc-ffff8e29e18b6800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:16:51 (1786756611) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:17:02 (1786756622) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:17:10 (1786756630) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:17:13 (1786756633) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:17:21 (1786756641) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 21:17:33 (1786756653) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:17:43 (1786756663) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:17:51 (1786756671) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:17:59 (1786756679) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:18:10 (1786756690) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:18:17 (1786756697) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.56867 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165567 s, 6.3 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:18:28 (1786756708) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.312903 s, 36.9 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:18:37 (1786756717) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:18:47 (1786756727) Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e29e18b6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e29e18b6800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e29e18b6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e29e18b6800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg356-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (70s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:19:58 (1786756798) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.231308 s, 18.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e29e18b6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e29e18b6800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:20:39 (1786756839) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e29e18b6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e29e18b6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e29e18b6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e29e18b6800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg356-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:22:01 (1786756921) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.19394 s, 16.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 14 21:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.849s user 0m0.000s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e29e18b6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e29e18b6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.009s user 0m0.001s sys 0m0.199s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e29e18b6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e29e18b6800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.147s user 0m0.000s sys 0m0.180s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg356-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e29e18b6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e29e18b6800.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:23:36 (1786757016) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:23:39 (1786757019) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:23:42 (1786757022) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:23:46 (1786757026) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 21:23:54 (1786757034) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.770638 s, 16.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:24:07 (1786757047) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379322 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03956 s, 26.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:24:15 (1786757055) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:24:24 (1786757064) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00359296 s, 285 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00098719 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00516566 s, 198 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00128803 s, 795 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 21:24:35 (1786757075) create test files with size 15733584 .. ae3d0b2570414c0e2eec9c82a94daa03 - 822b85d366a063050486891ee35bf262 - 2a05746fa9544f23890dba4752cadd5e - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg356-client.virtnet /mnt/lustre (opts:) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 21:25:00 (1786757100) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:25:15 (1786757115) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:25:22 (1786757122) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107093 s, 9.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283012 s, 37.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 21:25:31 (1786757131) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0804912 s, 26.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.304542 s, 13.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139451 s, 15.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 21:25:47 (1786757147) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 21:25:51 (1786757151) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:25:55 (1786757155) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582146 s, 18.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 21:26:04 (1786757164) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.404311 s, 7.8 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:26:14 (1786757174) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:26:26 (1786757186) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.455045 s, 23.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg356-client.virtnet /mnt/lustre (opts:) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:26:40 (1786757200) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8e29c5462800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.413722 s, 25.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg356-client.virtnet /mnt/lustre (opts:) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8e29c9383000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:26:59 (1786757219) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:27:38 (1786757258) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:28:10 (1786757290) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 21:28:13 (1786757293) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.334621 s, 9.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.111893 s, 28.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa391bc28 (nosync,stale) Mirror 2: 0xa391bc28 (nosync,stale) ** verify mirror 2 doesn't change original checksum: acc0ea8ec6b3ba3f36424174b19e52bf - mirror 1 checksum: 6f1d0ddcfb12c5a4ef0f69e84095d089 - mirror 2 checksum: acc0ea8ec6b3ba3f36424174b19e52bf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T21:28:15-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-14T21:28:15-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: acc0ea8ec6b3ba3f36424174b19e52bf - mirror 1 checksum: 6f1d0ddcfb12c5a4ef0f69e84095d089 - mirror 2 checksum: 6f1d0ddcfb12c5a4ef0f69e84095d089 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T21:28:15-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-14T21:28:15-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:28:23 (1786757303) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568796 s, 18.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:28:30 (1786757310) Failing mds1 on oleg356-server Stopping /mnt/lustre-mds1 (opts:) on oleg356-server 21:28:43 (1786757323) shut down facet: mds1 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds1 to oleg356-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 21:29:18 (1786757358) targets are mounted 21:29:18 (1786757358) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490748 s, 21.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:29:36 (1786757376) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0552226 s, 371 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0250027 s, 819 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:29:47 (1786757387) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0480195 s, 426 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0142433 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:29:59 (1786757399) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0106596 s, 1.5 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:30:08 (1786757408) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0433841 s, 472 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:30:17 (1786757417) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00443404 s, 924 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0277362 s, 148 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:30:25 (1786757425) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (30s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:30:55 (1786757455) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:31:24 (1786757484) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:31:48 (1786757508) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:32:12 (1786757532) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 21:32:22 (1786757542) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57571 mirror split pid 57572 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57571 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57572 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 21:33:31 (1786757611) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 7614 skipping zero size read skipping zero size read fallocating to largest ever: 0x156bf 000003 1786757614.545914 fallocate 0x00f4f4 thru 0x0156be (0x61cb bytes) fallocating to largest ever: 0x383d5 000004 1786757614.562943 fallocate 0x02b54f thru 0x0383d4 (0xce86 bytes) 000005 1786757614.577559 punch from 0x00660d to 0x00fd79 (0x976d bytes) 000006 1786757614.591008 punch from 0x00d443 to 0x00f4fe (0x20bc bytes) 000009 1786757614.598855 mapread 0x009832 thru 0x0156be (0xbe8d bytes) fallocating to largest ever: 0x40000 000010 1786757614.641722 fallocate 0x034879 thru 0x03ffff (0xb787 bytes) 000011 1786757614.648642 read 0x01e277 thru 0x028ad2 (0xa85c bytes) 000012 1786757614.668116 write 0x0368c9 thru 0x036f6d (0x6a5 bytes) 000013 1786757614.677239 punch from 0x007567 to 0x014a98 (0xd532 bytes) 000014 1786757614.689000 mapwrite 0x011f7e thru 0x0150d6 (0x3159 bytes) 000015 1786757614.754134 trunc from 0x03ed16 to 0x03ffff (0x12ea bytes) 000016 1786757614.764797 write 0x03f658 thru 0x03ffff (0x9a8 bytes) 000017 1786757614.784027 MIRROR_EXTEND to 2 mirrors 000018 1786757616.200289 punch from 0x010715 to 0x013244 (0x2b30 bytes) 000019 1786757616.238576 mapwrite 0x0184ad thru 0x0188fd (0x451 bytes) 000020 1786757616.260156 write 0x035feb thru 0x039669 (0x367f bytes) 000021 1786757616.286993 trunc from 0x0113f0 to 0x01c4ae (0xb0bf bytes) 000022 1786757616.293625 trunc from 0x040000 to 0x029bd4 (0x1642b bytes) 000023 1786757616.338209 punch from 0x00e2ff to 0x01c1ab (0xdead bytes) 000024 1786757616.346173 mapread 0x020a8d thru 0x0297e0 (0x8d54 bytes) 000025 1786757616.385767 trunc from 0x02170d to 0x0273af (0x5ca3 bytes) 000026 1786757616.395487 trunc from 0x029bd5 to 0x017f1d (0x11cb7 bytes) 000027 1786757616.413383 write 0x032c46 thru 0x03c595 (0x9950 bytes) 000029 1786757616.417970 mapread 0x017bc9 thru 0x026267 (0xe69f bytes) 000030 1786757616.434297 write 0x01a794 thru 0x01d79a (0x3007 bytes) 000031 1786757616.435539 write 0x001f9e thru 0x003458 (0x14bb bytes) 000032 1786757616.444938 mapread 0x02b405 thru 0x0311b3 (0x5daf bytes) 000033 1786757616.451673 write 0x00249d thru 0x010df3 (0xe957 bytes) 000034 1786757616.456522 write 0x00607f thru 0x0119ef (0xb971 bytes) 000035 1786757616.460349 write 0x00a50a thru 0x00ff9c (0x5a93 bytes) 000036 1786757616.463229 mapread 0x018820 thru 0x0212e9 (0x8aca bytes) 000038 1786757616.470609 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c596): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c596): Mirror 2: 0x7e2fe252 (stale) CRC-32 checksum value for chunk [0, 0x3c596): Mirror 1: 0xa4919570 Mirror 3: 0xa4919570 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c596: skipped 000039 1786757617.675010 mapread 0x03a7e1 thru 0x03c595 (0x1db5 bytes) 000040 1786757617.767981 read 0x024284 thru 0x02c6c2 (0x843f bytes) 000041 1786757617.810063 punch from 0x011ffa to 0x01cce7 (0xacee bytes) 000043 1786757617.859031 trunc from 0x03c596 to 0x00f830 (0x2cd65 bytes) 000044 1786757617.909983 read 0x008103 thru 0x00f830 (0x772e bytes) 000045 1786757617.927994 fallocate 0x00617f thru 0x009557 (0x33d9 bytes) 000047 1786757617.934580 fallocate 0x02883f thru 0x02e16e (0x5930 bytes) 000048 1786757617.954764 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xf831): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf831): Mirror 2: 0x1a025127 (stale) Mirror 3: 0x19e6feb8 (stale) CRC-32 checksum value for chunk [0, 0xf831): Mirror 1: 0x19e6feb8 Mirror 4: 0x19e6feb8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf831: skipped 000049 1786757619.016787 write 0x03524e thru 0x03cebc (0x7c6f bytes) 000050 1786757619.071544 trunc from 0x02257a to 0x023ac0 (0x1547 bytes) 000051 1786757619.081117 mapread 0x0163de thru 0x024d3e (0xe961 bytes) 000052 1786757619.131969 write 0x032808 thru 0x03538f (0x2b88 bytes) 000053 1786757619.152355 write 0x035b3e thru 0x036fcc (0x148f bytes) 000054 1786757619.153984 trunc from 0x03cebd to 0x00d975 (0x2f547 bytes) 000057 1786757619.198825 read 0x000257 thru 0x000831 (0x5db bytes) 000058 1786757619.224191 trunc from 0x005a7b to 0x00d975 (0x7efb bytes) 000059 1786757619.250679 punch from 0x00a03c to 0x00d975 (0x393a bytes) 000060 1786757619.264529 write 0x036515 thru 0x03ffff (0x9aeb bytes) 000061 1786757619.275533 MIRROR_RESYNC 4 mirrors 000062 1786757620.808207 write 0x036162 thru 0x03ffff (0x9e9e bytes) 000063 1786757620.856037 punch from 0x0271d8 to 0x0369a6 (0xf7cf bytes) 000064 1786757620.873487 trunc from 0x001a11 to 0x00bd0d (0xa2fd bytes) 000065 1786757620.879916 trunc from 0x026509 to 0x02fdca (0x98c2 bytes) 000066 1786757620.894578 punch from 0x0278f3 to 0x02e958 (0x7066 bytes) 000067 1786757620.907350 mapread 0x00b63b thru 0x018ef0 (0xd8b6 bytes) 000068 1786757620.989933 trunc from 0x01ee2c to 0x02ac88 (0xbe5d bytes) 000069 1786757621.000133 read 0x0256da thru 0x030823 (0xb14a bytes) 000071 1786757621.031255 mapwrite 0x020fae thru 0x023129 (0x217c bytes) 000072 1786757621.104147 write 0x02ed01 thru 0x0356f6 (0x69f6 bytes) 000073 1786757621.121984 mapwrite 0x03b408 thru 0x03ffff (0x4bf8 bytes) 000074 1786757621.158462 write 0x0242ff thru 0x029df1 (0x5af3 bytes) 000075 1786757621.167222 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x95248f11 (stale) Mirror 3: 0x95248f11 (stale) Mirror 4: 0x95248f11 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaff527dd Mirror 5: 0xaff527dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000076 1786757622.254687 punch from 0x0178b5 to 0x0214e0 (0x9c2c bytes) 000077 1786757622.305271 read 0x0210d7 thru 0x02e032 (0xcf5c bytes) 000078 1786757622.320156 punch from 0x011f8e to 0x01f282 (0xd2f5 bytes) 000079 1786757622.325029 mapwrite 0x032986 thru 0x03397f (0xffa bytes) 000081 1786757622.348022 punch from 0x00defe to 0x01cd42 (0xee45 bytes) 000082 1786757622.353579 fallocate 0x01ac06 thru 0x025a62 (0xae5d bytes) 000083 1786757622.358769 mapread 0x039007 thru 0x03e490 (0x548a bytes) 000084 1786757622.374377 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x95248f11 (stale) Mirror 4: 0x95248f11 (stale) Mirror 5: 0xaff527dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x95248f11 (stale) Mirror 4: 0x95248f11 (stale) Mirror 5: 0xaff527dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000085 1786757622.902263 trunc from 0x032c4d to 0x03caa4 (0x9e58 bytes) 000086 1786757622.911543 trunc from 0x040000 to 0x037604 (0x89fb bytes) 000087 1786757623.041336 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37605): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37605): Mirror 3: 0xa597e443 (stale) Mirror 4: 0xa597e443 (stale) Mirror 5: 0x6e6653a3 (stale) CRC-32 checksum value for chunk [0, 0x37605): Mirror 1: 0x60d9cf58 Mirror 6: 0x60d9cf58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37605: skipped 000089 1786757624.881147 mapwrite 0x011fcd thru 0x01d504 (0xb538 bytes) 000090 1786757625.128561 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37605): Mirror 4: 0xa597e443 (stale) Mirror 5: 0x6e6653a3 (stale) Mirror 6: 0x60d9cf58 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37605): Mirror 4: 0xa597e443 (stale) Mirror 5: 0x6e6653a3 (stale) Mirror 6: 0x60d9cf58 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37605: skipped 000091 1786757625.739020 mapread 0x007a72 thru 0x016c49 (0xf1d8 bytes) 000092 1786757625.774191 punch from 0x02eb70 to 0x034d20 (0x61b1 bytes) 000093 1786757625.781064 mapread 0x02d581 thru 0x037604 (0xa084 bytes) 000094 1786757625.794554 fallocate 0x02c4a5 thru 0x03bdd3 (0xf92f bytes) 000095 1786757625.799352 mapread 0x01e845 thru 0x02ce6b (0xe627 bytes) 000096 1786757625.810124 trunc from 0x019d68 to 0x01c3ff (0x2698 bytes) 000098 1786757625.816162 punch from 0x001ae5 to 0x003e84 (0x23a0 bytes) 000099 1786757625.823696 trunc from 0x0098fc to 0x015174 (0xb879 bytes) 000100 1786757625.833586 punch from 0x00b68a to 0x016c70 (0xb5e7 bytes) 000101 1786757625.842292 punch from 0x006be2 to 0x01603e (0xf45d bytes) 000102 1786757625.854576 punch from 0x01c8a1 to 0x024e25 (0x8585 bytes) 000103 1786757625.861925 mapwrite 0x02696d thru 0x02dc2d (0x72c1 bytes) 000104 1786757625.900638 trunc from 0x027274 to 0x02ef18 (0x7ca5 bytes) 000105 1786757625.914764 fallocate 0x00a056 thru 0x00a212 (0x1bd bytes) 000106 1786757625.920589 trunc from 0x017563 to 0x025a3a (0xe4d8 bytes) 000108 1786757625.939042 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37605): Mirror 5: 0x6e6653a3 (stale) Mirror 6: 0x60d9cf58 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37605): Mirror 5: 0x6e6653a3 (stale) Mirror 6: 0x60d9cf58 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37605: skipped 000109 1786757626.414182 trunc from 0x036593 to 0x037604 (0x1072 bytes) 000110 1786757626.425015 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37605): Mirror 6: 0x60d9cf58 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37605): Mirror 6: 0x60d9cf58 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37605: skipped 000112 1786757626.759752 punch from 0x02bd27 to 0x0310fe (0x53d8 bytes) 000113 1786757626.769194 write 0x002870 thru 0x0038ca (0x105b bytes) 000114 1786757626.813019 fallocate 0x01b4e5 thru 0x022ce7 (0x7803 bytes) 000115 1786757626.820145 trunc from 0x037605 to 0x0299cd (0xdc37 bytes) 000116 1786757626.855131 mapread 0x01c23f thru 0x01efa1 (0x2d63 bytes) 000117 1786757626.903276 trunc from 0x0299ce to 0x034f0f (0xb542 bytes) 000118 1786757626.949762 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34f10): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f10): Mirror 6: 0xed928942 (stale) CRC-32 checksum value for chunk [0, 0x34f10): Mirror 1: 0x250258f7 Mirror 7: 0x250258f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f10: skipped 000119 1786757628.605254 punch from 0x00a470 to 0x015f8b (0xbb1c bytes) 000120 1786757628.660716 punch from 0x01f051 to 0x020bfd (0x1bad bytes) 000121 1786757628.667535 fallocate 0x03f366 thru 0x03ffff (0xc9a bytes) 000122 1786757628.674597 mapwrite 0x0092e7 thru 0x01192b (0x8645 bytes) 000123 1786757628.717644 punch from 0x02221e to 0x02b31b (0x90fe bytes) 000124 1786757628.723858 trunc from 0x02573e to 0x02f4ed (0x9db0 bytes) 000125 1786757628.729709 punch from 0x0005fd to 0x001c5c (0x1660 bytes) 000126 1786757628.743436 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34f10): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34f10): Mirror 6: 0xed928942 (stale) Mirror 7: 0x250258f7 (stale) CRC-32 checksum value for chunk [0, 0x34f10): Mirror 1: 0x2f3bf1e5 Mirror 8: 0x2f3bf1e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34f10: skipped 000127 1786757630.367078 mapwrite 0x03e9b9 thru 0x03ffff (0x1647 bytes) 000128 1786757630.508772 write 0x00d053 thru 0x01a4d2 (0xd480 bytes) 000129 1786757630.533796 MIRROR_RESYNC 4 mirrors 000130 1786757631.759116 read 0x00f9e5 thru 0x01c0db (0xc6f7 bytes) 000131 1786757631.773565 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000132 1786757632.058294 trunc from 0x0302e8 to 0x03ca72 (0xc78b bytes) 000134 1786757632.156262 MIRROR_RESYNC 3 mirrors 000135 1786757633.478231 mapwrite 0x01ca57 thru 0x02acc9 (0xe273 bytes) 000136 1786757633.620059 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xd73af08b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xd73af08b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000137 1786757634.141651 mapwrite 0x003fde thru 0x0080f4 (0x4117 bytes) 000138 1786757634.191348 mapread 0x0085f0 thru 0x013864 (0xb275 bytes) 000139 1786757634.202484 trunc from 0x040000 to 0x003033 (0x3cfcc bytes) 000140 1786757634.231273 trunc from 0x0005d2 to 0x002e9a (0x28c9 bytes) 000142 1786757634.241843 trunc from 0x003034 to 0x02c20c (0x291d9 bytes) 000143 1786757634.267777 trunc from 0x02832d to 0x0289bd (0x691 bytes) 000144 1786757634.280206 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c20d): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c20d): Mirror 6: 0xfcc6291c (stale) CRC-32 checksum value for chunk [0, 0x2c20d): Mirror 1: 0x6346dd93 Mirror 7: 0x6346dd93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c20d: skipped 000146 1786757635.316956 mapread 0x0184a6 thru 0x01de23 (0x597e bytes) 000147 1786757635.395824 mapwrite 0x00f356 thru 0x0179c3 (0x866e bytes) 000148 1786757635.609329 read 0x01642d thru 0x01d1cd (0x6da1 bytes) 000149 1786757635.619696 read 0x0002cf thru 0x0077ce (0x7500 bytes) 000150 1786757635.627370 write 0x01d47e thru 0x02b360 (0xdee3 bytes) 000151 1786757635.636656 fallocate 0x008dd3 thru 0x015e1a (0xd048 bytes) 000152 1786757635.640944 trunc from 0x0136ae to 0x01ad09 (0x765c bytes) 000153 1786757635.647510 mapwrite 0x0004fe thru 0x00c6db (0xc1de bytes) 000154 1786757635.680246 fallocate 0x017fb2 thru 0x025b18 (0xdb67 bytes) 000155 1786757635.686521 trunc from 0x02ac8a to 0x02c20c (0x1583 bytes) 000156 1786757635.692303 fallocate 0x037bab thru 0x03f776 (0x7bcc bytes) 000157 1786757635.697792 read 0x022bff thru 0x026d2a (0x412c bytes) 000159 1786757635.703832 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f777): Mirror 6: 0xcc970172 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f777): Mirror 6: 0xcc970172 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f777: skipped 000160 1786757636.117206 fallocate 0x00f99b thru 0x013a4c (0x40b2 bytes) 000161 1786757636.130436 mapwrite 0x01f90f thru 0x02f742 (0xfe34 bytes) 000162 1786757636.290181 fallocate 0x00bbf7 thru 0x016431 (0xa83b bytes) 000163 1786757636.301409 write 0x030e07 thru 0x039296 (0x8490 bytes) 000165 1786757636.337827 read 0x03d847 thru 0x03f776 (0x1f30 bytes) 000166 1786757636.350673 mapwrite 0x03daa0 thru 0x03faac (0x200d bytes) 000168 1786757636.422160 mapwrite 0x034e39 thru 0x0391f0 (0x43b8 bytes) 000169 1786757636.455186 fallocate 0x0244c2 thru 0x03410a (0xfc49 bytes) 000170 1786757636.470517 mapread 0x02140f thru 0x0248d1 (0x34c3 bytes) 000171 1786757636.476020 mapwrite 0x00fec4 thru 0x01895e (0x8a9b bytes) 000172 1786757636.570798 mapwrite 0x02d558 thru 0x031d79 (0x4822 bytes) 000173 1786757636.623040 mapread 0x01e4ce thru 0x022eec (0x4a1f bytes) 000174 1786757636.627648 fallocate 0x026a68 thru 0x034233 (0xd7cc bytes) 000175 1786757636.639220 write 0x011a27 thru 0x017598 (0x5b72 bytes) 000176 1786757636.640445 read 0x01c7a2 thru 0x02a070 (0xd8cf bytes) 000177 1786757636.641102 punch from 0x0270df to 0x02c1d6 (0x50f8 bytes) 000178 1786757636.659475 mapread 0x018fcf thru 0x01cf7d (0x3faf bytes) 000179 1786757636.665789 trunc from 0x03faad to 0x000ad5 (0x3efd7 bytes) 000180 1786757636.709003 trunc from 0x00060a to 0x000ad5 (0x4cc bytes) 000181 1786757636.722975 read 0x00075d thru 0x000ad5 (0x379 bytes) 000182 1786757636.740339 mapread 0x000700 thru 0x000ad5 (0x3d6 bytes) 000185 1786757636.743340 read 0x0007ff thru 0x000ad5 (0x2d7 bytes) 000187 1786757636.748657 fallocate 0x03d2a6 thru 0x03eca9 (0x1a04 bytes) 000188 1786757636.763953 mapread 0x02918b thru 0x0376f2 (0xe568 bytes) 000189 1786757636.792696 mapread 0x023eeb thru 0x033137 (0xf24d bytes) 000190 1786757636.801073 mapwrite 0x012713 thru 0x01c693 (0x9f81 bytes) 000191 1786757636.853988 trunc from 0x023fde to 0x02e19f (0xa1c2 bytes) 000192 1786757636.863880 mapwrite 0x0179ad thru 0x021292 (0x98e6 bytes) 000195 1786757636.892658 mapwrite 0x025543 thru 0x02ceae (0x796c bytes) 000196 1786757637.009660 punch from 0x01c3aa to 0x029429 (0xd080 bytes) 000197 1786757637.017066 mapread 0x00d237 thru 0x011c7d (0x4a47 bytes) 000198 1786757637.018980 mapread 0x001c05 thru 0x0070f0 (0x54ec bytes) 000199 1786757637.021242 mapread 0x02137b thru 0x02b692 (0xa318 bytes) 000200 1786757637.080799 fallocate 0x011a07 thru 0x012469 (0xa63 bytes) 000201 1786757637.087935 trunc from 0x03ecaa to 0x00d008 (0x31ca1 bytes) 000202 1786757637.114792 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xd009): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd009): Mirror 6: 0x46de49c4 (stale) CRC-32 checksum value for chunk [0, 0xd009): Mirror 1: 0x3328339f Mirror 7: 0x3328339f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd009: skipped 000203 1786757638.055458 punch from 0x00afec to 0x00d008 (0x201d bytes) 000204 1786757638.097798 write 0x033620 thru 0x03bae6 (0x84c7 bytes) 000205 1786757638.105092 read 0x01d14b thru 0x01e87a (0x1730 bytes) 000206 1786757638.110128 punch from 0x016083 to 0x01c84e (0x67cc bytes) 000207 1786757638.115167 mapwrite 0x02597c thru 0x029cb7 (0x433c bytes) 000208 1786757638.148285 trunc from 0x0317e3 to 0x03bae6 (0xa304 bytes) 000209 1786757638.154083 punch from 0x00fecf to 0x014c2d (0x4d5f bytes) 000210 1786757638.158905 trunc from 0x00a4c3 to 0x00b19b (0xcd9 bytes) 000211 1786757638.163172 punch from 0x02f0d6 to 0x03890a (0x9835 bytes) 000212 1786757638.168003 mapread 0x01ed5c thru 0x024a13 (0x5cb8 bytes) 000213 1786757638.186144 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bae7): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bae7): Mirror 6: 0x94ef1d09 (stale) Mirror 7: 0x5111500f (stale) CRC-32 checksum value for chunk [0, 0x3bae7): Mirror 1: 0x36516b73 Mirror 8: 0x36516b73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bae7: skipped 000214 1786757638.797371 read 0x01f2bb thru 0x026121 (0x6e67 bytes) 000215 1786757638.818730 read 0x0156ef thru 0x0190bd (0x39cf bytes) 000216 1786757638.856015 write 0x00903c thru 0x009519 (0x4de bytes) 000217 1786757638.902722 mapread 0x026bb3 thru 0x032d1a (0xc168 bytes) 000218 1786757638.964640 write 0x012ede thru 0x01eb2a (0xbc4d bytes) 000219 1786757638.975684 mapread 0x018c2a thru 0x02613e (0xd515 bytes) 000220 1786757639.011831 punch from 0x00a08d to 0x016e29 (0xcd9d bytes) 000221 1786757639.029333 mapread 0x03b9ef thru 0x03bae6 (0xf8 bytes) 000222 1786757639.036923 mapwrite 0x00721b thru 0x007541 (0x327 bytes) 000223 1786757639.053379 write 0x005604 thru 0x005d96 (0x793 bytes) 000224 1786757639.058296 read 0x010a80 thru 0x01ee6d (0xe3ee bytes) 000225 1786757639.066850 read 0x0369b1 thru 0x03bae6 (0x5136 bytes) 000226 1786757639.067870 trunc from 0x008ce7 to 0x00a6a3 (0x19bd bytes) 000227 1786757639.072384 trunc from 0x03bae7 to 0x00a3db (0x3170b bytes) 000228 1786757639.117632 fallocate 0x011bff thru 0x0199f6 (0x7df8 bytes) 000229 1786757639.122828 mapread 0x007948 thru 0x00a3db (0x2a94 bytes) 000230 1786757639.129362 fallocate 0x01a3f8 thru 0x02224b (0x7e54 bytes) 000231 1786757639.134104 mapwrite 0x03781b thru 0x03c3c7 (0x4bad bytes) 000232 1786757639.190911 fallocate 0x0231c8 thru 0x023fa9 (0xde2 bytes) 000233 1786757639.196144 fallocate 0x03f993 thru 0x03ffff (0x66d bytes) 000234 1786757639.202027 read 0x018982 thru 0x024e51 (0xc4d0 bytes) 000236 1786757639.203331 trunc from 0x03c3c8 to 0x024984 (0x17a43 bytes) 000237 1786757639.220382 trunc from 0x01d9ba to 0x023ada (0x6121 bytes) 000238 1786757639.226847 punch from 0x022d9e to 0x024984 (0x1be7 bytes) 000240 1786757639.235414 punch from 0x0161f0 to 0x0213cb (0xb1dc bytes) 000241 1786757639.241178 trunc from 0x010968 to 0x019ba4 (0x923d bytes) 000242 1786757639.246764 punch from 0x001c3c to 0x008dc3 (0x7188 bytes) 000243 1786757639.251857 mapwrite 0x0235a5 thru 0x030667 (0xd0c3 bytes) 000244 1786757639.333027 trunc from 0x030668 to 0x019965 (0x16d02 bytes) 000245 1786757639.351308 fallocate 0x00c61d thru 0x00ec82 (0x2666 bytes) 000246 1786757639.356908 trunc from 0x019966 to 0x03f2ea (0x25985 bytes) 000247 1786757639.377197 write 0x002aa5 thru 0x00e6a2 (0xbbfe bytes) 000248 1786757639.382648 fallocate 0x00d545 thru 0x018738 (0xb1f4 bytes) 000249 1786757639.394469 read 0x0381e7 thru 0x03f2ea (0x7104 bytes) 000250 1786757639.412143 mapread 0x03cb0a thru 0x03f2ea (0x27e1 bytes) 000251 1786757639.419060 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f2eb): Mirror 7: 0x75b9d522 (stale) Mirror 8: 0xe4ca2127 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2eb): Mirror 7: 0x75b9d522 (stale) Mirror 8: 0xe4ca2127 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2eb: skipped 000252 1786757639.860255 read 0x00fc64 thru 0x01dd7b (0xe118 bytes) 000253 1786757639.883916 fallocate 0x01a442 thru 0x01c2fa (0x1eb9 bytes) 000254 1786757639.898232 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f2eb): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2eb): Mirror 7: 0x75b9d522 (stale) Mirror 8: 0xe4ca2127 (stale) CRC-32 checksum value for chunk [0, 0x3f2eb): Mirror 1: 0xb4228cfe Mirror 9: 0xb4228cfe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2eb: skipped 000255 1786757641.263312 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f2eb): Mirror 8: 0xe4ca2127 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2eb): Mirror 8: 0xe4ca2127 (stale) CRC-32 checksum value for chunk [0, 0x3f2eb): Mirror 1: 0xb4228cfe Mirror 9: 0xb4228cfe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2eb: skipped 000256 1786757641.821350 trunc from 0x02a91d to 0x030c59 (0x633d bytes) 000257 1786757641.871579 fallocate 0x03faff thru 0x03ffff (0x501 bytes) 000258 1786757641.877714 mapread 0x01893f thru 0x01cea2 (0x4564 bytes) 000259 1786757641.896108 fallocate 0x02fb24 thru 0x036e1c (0x72f9 bytes) 000260 1786757641.901473 read 0x000521 thru 0x009b2e (0x960e bytes) 000261 1786757641.909270 write 0x01c59f thru 0x01f6d1 (0x3133 bytes) 000262 1786757641.914496 trunc from 0x00103e to 0x009d05 (0x8cc8 bytes) 000263 1786757641.921103 read 0x0093eb thru 0x00c3cd (0x2fe3 bytes) 000264 1786757641.940208 trunc from 0x03f2eb to 0x023ce6 (0x1b604 bytes) 000265 1786757641.971555 mapread 0x0127d8 thru 0x01d34b (0xab74 bytes) 000266 1786757641.974148 mapwrite 0x0055f4 thru 0x00a689 (0x5096 bytes) 000267 1786757641.997911 MIRROR_RESYNC 3 mirrors 000268 1786757643.212551 trunc from 0x023ce7 to 0x032300 (0xe61a bytes) 000270 1786757643.262801 read 0x02b796 thru 0x0304ab (0x4d16 bytes) 000271 1786757643.273581 punch from 0x01907d to 0x0232b2 (0xa236 bytes) 000272 1786757643.279712 trunc from 0x032301 to 0x0203f2 (0x11f0e bytes) 000273 1786757643.320911 punch from 0x0030b8 to 0x00dd9d (0xace6 bytes) 000274 1786757643.337026 mapread 0x01e9ed thru 0x0203f2 (0x1a06 bytes) 000275 1786757643.365875 punch from 0x019aed to 0x0203f2 (0x6906 bytes) 000276 1786757643.378759 mapread 0x00f6d2 thru 0x0180ef (0x8a1e bytes) 000277 1786757643.414115 write 0x031f56 thru 0x03b3c4 (0x946f bytes) 000278 1786757643.416931 punch from 0x01bf61 to 0x023c10 (0x7cb0 bytes) 000279 1786757643.424036 write 0x03f751 thru 0x03ffff (0x8af bytes) 000280 1786757643.425630 punch from 0x011967 to 0x019c4b (0x82e5 bytes) 000281 1786757643.431950 fallocate 0x01a586 thru 0x025c10 (0xb68b bytes) 000282 1786757643.439948 trunc from 0x040000 to 0x01eb31 (0x214ce bytes) 000283 1786757643.467841 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1eb32): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eb32): Mirror 8: 0xf43eab3b (stale) Mirror 9: 0xf43eab3b (stale) CRC-32 checksum value for chunk [0, 0x1eb32): Mirror 1: 0xb2e212d2 Mirror 10: 0xb2e212d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eb32: skipped 000284 1786757644.453211 trunc from 0x01eb32 to 0x02f4a5 (0x10974 bytes) 000285 1786757644.604964 trunc from 0x02f4a6 to 0x02243d (0xd068 bytes) 000286 1786757644.627749 read 0x00b1a3 thru 0x015705 (0xa563 bytes) 000287 1786757644.636436 write 0x02fb12 thru 0x0317be (0x1cad bytes) 000288 1786757644.637614 fallocate 0x0162f2 thru 0x0238b2 (0xd5c1 bytes) 000291 1786757644.644359 write 0x005b08 thru 0x006fc0 (0x14b9 bytes) 000292 1786757644.653276 mapread 0x027900 thru 0x02b648 (0x3d49 bytes) 000294 1786757644.684808 punch from 0x02f54e to 0x0317be (0x2271 bytes) 000295 1786757644.717784 read 0x01bf7a thru 0x01d74b (0x17d2 bytes) 000297 1786757644.745947 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x317bf): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x317bf): Mirror 8: 0x4747cd97 (stale) Mirror 9: 0x4747cd97 (stale) Mirror 10: 0xea881a76 (stale) CRC-32 checksum value for chunk [0, 0x317bf): Mirror 1: 0xe53a049e Mirror 11: 0xe53a049e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x317bf: skipped 000298 1786757646.007276 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x317bf): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x317bf): Mirror 8: 0x4747cd97 (stale) Mirror 9: 0x4747cd97 (stale) Mirror 10: 0xea881a76 (stale) CRC-32 checksum value for chunk [0, 0x317bf): Mirror 1: 0xe53a049e Mirror 11: 0xe53a049e Mirror 12: 0xe53a049e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x317bf: skipped 000299 1786757647.211655 mapwrite 0x01efb8 thru 0x020b16 (0x1b5f bytes) 000300 1786757647.305087 trunc from 0x0317bf to 0x032adb (0x131d bytes) 000301 1786757647.327905 mapwrite 0x00a9ff thru 0x015404 (0xaa06 bytes) 000302 1786757647.361930 mapwrite 0x02b260 thru 0x02e928 (0x36c9 bytes) 000304 1786757647.382972 read 0x00186f thru 0x0099bc (0x814e bytes) 000305 1786757647.387894 mapwrite 0x03bb65 thru 0x03dde7 (0x2283 bytes) 000306 1786757647.428853 mapwrite 0x01c788 thru 0x0234d6 (0x6d4f bytes) 000307 1786757647.448143 fallocate 0x01ffb1 thru 0x02b29d (0xb2ed bytes) 000308 1786757647.455004 punch from 0x01c1d1 to 0x02bdea (0xfc1a bytes) 000309 1786757647.460058 trunc from 0x017b49 to 0x022417 (0xa8cf bytes) 000310 1786757647.463887 mapwrite 0x006b00 thru 0x00baf4 (0x4ff5 bytes) 000311 1786757647.479195 mapread 0x01d338 thru 0x02cb73 (0xf83c bytes) 000312 1786757647.502913 punch from 0x038224 to 0x038710 (0x4ed bytes) 000313 1786757647.506995 trunc from 0x03dde8 to 0x02b321 (0x12ac6 bytes) 000314 1786757647.526069 trunc from 0x02b322 to 0x004c80 (0x266a1 bytes) 000315 1786757647.541115 write 0x0362d8 thru 0x03ffff (0x9d28 bytes) 000316 1786757647.542782 trunc from 0x040000 to 0x0397c1 (0x683e bytes) 000317 1786757647.566367 read 0x01960f thru 0x026cf9 (0xd6eb bytes) 000318 1786757647.580466 punch from 0x025640 to 0x032ade (0xd49f bytes) 000319 1786757647.585015 read 0x0327fb thru 0x0397c1 (0x6fc7 bytes) 000320 1786757647.600180 read 0x02ec1f thru 0x0397c1 (0xaba3 bytes) 000321 1786757647.601031 mapwrite 0x025d1e thru 0x034463 (0xe746 bytes) 000322 1786757647.626760 read 0x018369 thru 0x019b02 (0x179a bytes) 000323 1786757647.627447 mapread 0x00c38a thru 0x00e8f0 (0x2567 bytes) 000324 1786757647.629273 fallocate 0x022018 thru 0x02b2d0 (0x92b9 bytes) 000325 1786757647.634269 fallocate 0x0029f5 thru 0x00e7a8 (0xbdb4 bytes) 000326 1786757647.643457 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x397c2): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397c2): Mirror 8: 0xb4fa94b1 (stale) Mirror 9: 0xb4fa94b1 (stale) Mirror 10: 0x59ec25a9 (stale) Mirror 11: 0xf54caccb (stale) Mirror 12: 0xf54caccb (stale) CRC-32 checksum value for chunk [0, 0x397c2): Mirror 1: 0xda489e3a Mirror 13: 0xda489e3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397c2: skipped 000327 1786757648.659560 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x397c2): Mirror 9: 0xb4fa94b1 (stale) Mirror 10: 0x59ec25a9 (stale) Mirror 11: 0xf54caccb (stale) Mirror 12: 0xf54caccb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397c2): Mirror 9: 0xb4fa94b1 (stale) Mirror 10: 0x59ec25a9 (stale) Mirror 11: 0xf54caccb (stale) Mirror 12: 0xf54caccb (stale) CRC-32 checksum value for chunk [0, 0x397c2): Mirror 1: 0xda489e3a Mirror 13: 0xda489e3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397c2: skipped 000328 1786757649.590543 trunc from 0x0397c2 to 0x02d11a (0xc6a7 bytes) 000330 1786757649.665888 mapread 0x027843 thru 0x02c7cd (0x4f8b bytes) 000331 1786757649.684816 read 0x0170ba thru 0x01b25e (0x41a5 bytes) 000332 1786757649.693867 write 0x00aee3 thru 0x00fca0 (0x4dbe bytes) 000333 1786757649.704272 read 0x015bb1 thru 0x01e57f (0x89cf bytes) 000334 1786757649.715215 read 0x018de4 thru 0x027ec0 (0xf0dd bytes) 000335 1786757649.716201 trunc from 0x02d11b to 0x014aaf (0x1866b bytes) 000336 1786757649.738585 write 0x02828e thru 0x03616f (0xdee2 bytes) 000338 1786757649.741599 mapwrite 0x010ccd thru 0x01bd33 (0xb067 bytes) 000339 1786757649.787673 trunc from 0x036170 to 0x01885f (0x1d910 bytes) 000340 1786757649.812522 MIRROR_RESYNC 6 mirrors 000341 1786757651.103008 trunc from 0x018860 to 0x017e2a (0xa35 bytes) 000342 1786757651.141209 punch from 0x00a2eb to 0x01221a (0x7f30 bytes) 000343 1786757651.152434 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x17e2b): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17e2b): Mirror 9: 0x2170a54b (stale) Mirror 10: 0x2170a54b (stale) Mirror 11: 0x2170a54b (stale) Mirror 12: 0x2170a54b (stale) Mirror 13: 0x2170a54b (stale) CRC-32 checksum value for chunk [0, 0x17e2b): Mirror 1: 0xe34961bc Mirror 14: 0xe34961bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17e2b: skipped 000344 1786757652.217465 write 0x016641 thru 0x0213ce (0xad8e bytes) 000345 1786757652.270850 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x213cf): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x213cf): Mirror 9: 0x45d7993f (stale) Mirror 10: 0x45d7993f (stale) Mirror 11: 0x45d7993f (stale) Mirror 12: 0x45d7993f (stale) Mirror 13: 0x45d7993f (stale) Mirror 14: 0x7655c6f3 (stale) CRC-32 checksum value for chunk [0, 0x213cf): Mirror 1: 0x5ac3131f Mirror 15: 0x5ac3131f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x213cf: skipped 000346 1786757653.561516 fallocate 0x03915f thru 0x03dcae (0x4b50 bytes) 000347 1786757653.596702 trunc from 0x03dcaf to 0x017781 (0x2652d bytes) 000348 1786757653.625561 write 0x0249e7 thru 0x0278e2 (0x2efc bytes) 000349 1786757653.630972 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x278e3): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278e3): Mirror 9: 0x2fef354f (stale) Mirror 10: 0x2fef354f (stale) Mirror 11: 0x2fef354f (stale) Mirror 12: 0x2fef354f (stale) Mirror 13: 0x2fef354f (stale) Mirror 14: 0x21013afc (stale) Mirror 15: 0x658207ce (stale) CRC-32 checksum value for chunk [0, 0x278e3): Mirror 1: 0xf249f309 Mirror 16: 0xf249f309 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278e3: skipped 000350 1786757654.569948 trunc from 0x0278e3 to 0x011ab3 (0x15e2f bytes) 000351 1786757654.614886 fallocate 0x00f739 thru 0x01cd80 (0xd648 bytes) 000352 1786757654.619108 punch from 0x012c6e to 0x013f4d (0x12e0 bytes) 000353 1786757654.624684 write 0x00a0ef thru 0x00ad6a (0xc7c bytes) 000354 1786757654.633485 read 0x00d4d7 thru 0x01492b (0x7455 bytes) 000356 1786757654.662308 trunc from 0x01cd81 to 0x028240 (0xb4c0 bytes) 000357 1786757654.683867 mapwrite 0x0335e3 thru 0x039f13 (0x6931 bytes) 000359 1786757654.739866 MIRROR_RESYNC 9 mirrors 000360 1786757656.863383 punch from 0x010960 to 0x014d91 (0x4432 bytes) 000361 1786757656.925159 mapwrite 0x00c10f thru 0x0171a8 (0xb09a bytes) 000363 1786757656.983065 trunc from 0x039f14 to 0x00216a (0x37da9 bytes) 000365 1786757657.014610 write 0x02d039 thru 0x034514 (0x74dc bytes) 000366 1786757657.018340 trunc from 0x034515 to 0x01af76 (0x1959e bytes) 000367 1786757657.038568 mapwrite 0x025559 thru 0x033946 (0xe3ee bytes) 000368 1786757657.106644 read 0x018a2a thru 0x01f7c9 (0x6da0 bytes) 000369 1786757657.124358 mapwrite 0x02164a thru 0x02eb2f (0xd4e6 bytes) 000370 1786757657.175488 read 0x004041 thru 0x0078e2 (0x38a2 bytes) 000371 1786757657.197723 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x33947): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33947): Mirror 9: 0xd662d3ea (stale) Mirror 10: 0xd662d3ea (stale) Mirror 11: 0xd662d3ea (stale) Mirror 12: 0xd662d3ea (stale) Mirror 13: 0xd662d3ea (stale) Mirror 14: 0xd662d3ea (stale) Mirror 15: 0xd662d3ea (stale) Mirror 16: 0xd662d3ea (stale) CRC-32 checksum value for chunk [0, 0x33947): Mirror 1: 0xc5ef1676 Mirror 17: 0xc5ef1676 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33947: skipped 000372 1786757658.096847 trunc from 0x033947 to 0x023f98 (0xf9ae bytes) 000373 1786757658.281755 trunc from 0x023f99 to 0x024b6c (0xbd4 bytes) 000374 1786757658.332055 mapread 0x00e47a thru 0x017fa6 (0x9b2d bytes) 000376 1786757658.386345 trunc from 0x024b6d to 0x0047b5 (0x203b7 bytes) 000377 1786757658.425961 trunc from 0x004706 to 0x0047b5 (0xb0 bytes) 000378 1786757658.431883 trunc from 0x00055d to 0x0047b5 (0x4259 bytes) 000379 1786757658.455106 MIRROR_RESYNC 10 mirrors 000380 1786757659.728121 punch from 0x003aa0 to 0x0047b5 (0xd16 bytes) 000381 1786757659.761941 write 0x0026d8 thru 0x011c62 (0xf58b bytes) 000382 1786757659.768486 mapwrite 0x02c7a8 thru 0x030698 (0x3ef1 bytes) 000383 1786757659.829818 trunc from 0x0026d1 to 0x01124a (0xeb7a bytes) 000384 1786757659.834803 mapread 0x000775 thru 0x00fb59 (0xf3e5 bytes) 000385 1786757659.844499 trunc from 0x030699 to 0x003c44 (0x2ca54 bytes) 000386 1786757659.863977 trunc from 0x002c16 to 0x003c44 (0x102f bytes) 000387 1786757659.869166 trunc from 0x001a5c to 0x003c44 (0x21e9 bytes) 000388 1786757659.874972 mapread 0x00022e thru 0x003c44 (0x3a17 bytes) 000389 1786757659.881646 mapread 0x00222d thru 0x003c44 (0x1a18 bytes) 000390 1786757659.883178 fallocate 0x004766 thru 0x008f33 (0x47ce bytes) 000391 1786757659.888766 mapread 0x00464b thru 0x008f33 (0x48e9 bytes) 000392 1786757659.898075 write 0x02893a thru 0x0374de (0xeba5 bytes) 000393 1786757659.901163 trunc from 0x0374df to 0x02ffad (0x7531 bytes) 000394 1786757659.915721 read 0x028b7f thru 0x02a655 (0x1ad7 bytes) 000396 1786757659.917251 punch from 0x0123db to 0x01a6a1 (0x82c7 bytes) 000398 1786757659.923777 punch from 0x026f8a to 0x02ffad (0x9024 bytes) 000399 1786757659.940135 write 0x00848d thru 0x00ff93 (0x7b07 bytes) 000400 1786757659.946364 read 0x028af3 thru 0x0290d0 (0x5de bytes) 000402 1786757659.958085 mapread 0x019687 thru 0x01a4aa (0xe24 bytes) 000403 1786757659.959131 mapread 0x00011e thru 0x00c53e (0xc421 bytes) 000404 1786757659.962104 read 0x017e38 thru 0x01f88a (0x7a53 bytes) 000405 1786757659.963086 read 0x022c25 thru 0x02ad54 (0x8130 bytes) 000406 1786757659.964670 trunc from 0x0166b7 to 0x01909f (0x29e9 bytes) 000407 1786757659.970182 punch from 0x005883 to 0x0063fe (0xb7c bytes) 000408 1786757659.975957 fallocate 0x01f8f2 thru 0x02188e (0x1f9d bytes) 000409 1786757659.981924 write 0x026c81 thru 0x02ad2a (0x40aa bytes) 000410 1786757659.986575 MIRROR_RESYNC 10 mirrors 000411 1786757662.933423 mapwrite 0x0263fe thru 0x026df7 (0x9fa bytes) 000413 1786757663.012042 mapread 0x01d6e8 thru 0x01f375 (0x1c8e bytes) 000414 1786757663.027674 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ffae): Mirror 9: 0xc29c086d (stale) Mirror 11: 0xc29c086d (stale) Mirror 12: 0xc29c086d (stale) Mirror 13: 0xc29c086d (stale) Mirror 14: 0xc29c086d (stale) Mirror 15: 0xc29c086d (stale) Mirror 16: 0xc29c086d (stale) Mirror 17: 0xc29c086d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffae): Mirror 9: 0xc29c086d (stale) Mirror 11: 0xc29c086d (stale) Mirror 12: 0xc29c086d (stale) Mirror 13: 0xc29c086d (stale) Mirror 14: 0xc29c086d (stale) Mirror 15: 0xc29c086d (stale) Mirror 16: 0xc29c086d (stale) Mirror 17: 0xc29c086d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffae: skipped 000415 1786757663.626191 mapread 0x01b2f0 thru 0x01bbd0 (0x8e1 bytes) 000416 1786757663.635905 mapread 0x00ed1c thru 0x01d20f (0xe4f4 bytes) 000417 1786757663.668918 trunc from 0x0267d6 to 0x02ffad (0x97d8 bytes) 000418 1786757663.678311 trunc from 0x02ffae to 0x01ec3c (0x11371 bytes) 000419 1786757663.707096 trunc from 0x01ec3d to 0x006c1f (0x1801d bytes) 000420 1786757663.722958 trunc from 0x006c20 to 0x020890 (0x19c71 bytes) 000421 1786757663.741210 write 0x01b971 thru 0x0204bc (0x4b4c bytes) 000422 1786757663.749920 trunc from 0x020891 to 0x01e3d2 (0x24be bytes) 000423 1786757663.766052 fallocate 0x038a3b thru 0x03ffff (0x75c5 bytes) 000424 1786757663.770888 fallocate 0x03b32a thru 0x03ffff (0x4cd6 bytes) 000425 1786757663.776034 trunc from 0x040000 to 0x031fb0 (0xe04f bytes) 000426 1786757663.789954 mapread 0x02e57c thru 0x031fb0 (0x3a35 bytes) 000428 1786757663.801294 mapwrite 0x011eb1 thru 0x014d2c (0x2e7c bytes) 000429 1786757663.827902 read 0x014a5c thru 0x0219ef (0xcf94 bytes) 000430 1786757663.837097 write 0x0195f4 thru 0x01c4db (0x2ee8 bytes) 000431 1786757663.838447 read 0x02e0b6 thru 0x02f536 (0x1481 bytes) 000433 1786757663.840125 read 0x01a49a thru 0x0279c1 (0xd528 bytes) 000434 1786757663.853739 trunc from 0x031fb1 to 0x000803 (0x317ad bytes) 000435 1786757663.872201 read 0x0001e8 thru 0x000803 (0x61c bytes) 000436 1786757663.872968 mapwrite 0x03a55f thru 0x03e9c3 (0x4465 bytes) 000437 1786757663.932975 MIRROR_RESYNC 9 mirrors 000438 1786757665.831209 write 0x029bad thru 0x02a29a (0x6ee bytes) 000440 1786757665.934675 trunc from 0x005037 to 0x00db7e (0x8b48 bytes) 000441 1786757665.960143 fallocate 0x00d756 thru 0x013a78 (0x6323 bytes) 000443 1786757665.972276 read 0x00e437 thru 0x010596 (0x2160 bytes) 000444 1786757665.978699 trunc from 0x03e9c4 to 0x035a2d (0x8f96 bytes) 000445 1786757666.017776 fallocate 0x00e507 thru 0x01c6de (0xe1d8 bytes) 000446 1786757666.028968 punch from 0x0349f8 to 0x035a2d (0x1036 bytes) 000447 1786757666.036122 read 0x01224f thru 0x0203b8 (0xe16a bytes) 000448 1786757666.055934 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x35a2e): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35a2e): Mirror 9: 0xaa6fea12 (stale) Mirror 11: 0xaa6fea12 (stale) Mirror 12: 0xaa6fea12 (stale) Mirror 13: 0xaa6fea12 (stale) Mirror 14: 0xaa6fea12 (stale) Mirror 15: 0xaa6fea12 (stale) Mirror 16: 0xaa6fea12 (stale) Mirror 17: 0xaa6fea12 (stale) CRC-32 checksum value for chunk [0, 0x35a2e): Mirror 1: 0x1ff2025b Mirror 18: 0x1ff2025b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35a2e: skipped 000449 1786757667.599080 fallocate 0x022238 thru 0x02fe3c (0xdc05 bytes) 000450 1786757667.676299 trunc from 0x00a2af to 0x011101 (0x6e53 bytes) 000451 1786757667.685239 trunc from 0x035a2e to 0x01cf0b (0x18b22 bytes) 000452 1786757667.731134 MIRROR_RESYNC 10 mirrors 000453 1786757670.012212 mapread 0x004867 thru 0x0092c3 (0x4a5d bytes) 000454 1786757670.033306 punch from 0x004ef3 to 0x00e184 (0x9292 bytes) 000455 1786757670.140278 write 0x008027 thru 0x00eef1 (0x6ecb bytes) 000457 1786757670.165324 read 0x011d00 thru 0x01698a (0x4c8b bytes) 000459 1786757670.195340 read 0x01997b thru 0x01cf0b (0x3591 bytes) 000460 1786757670.230619 trunc from 0x00ed1f to 0x017e9b (0x917d bytes) 000461 1786757670.282928 read 0x00e416 thru 0x01188d (0x3478 bytes) 000462 1786757670.308277 mapwrite 0x008262 thru 0x00cda3 (0x4b42 bytes) 000464 1786757670.340976 punch from 0x01b4ca to 0x01cf0b (0x1a42 bytes) 000465 1786757670.367759 mapwrite 0x008ebb thru 0x014d4d (0xbe93 bytes) 000466 1786757670.420225 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cf0c): Mirror 9: 0x5c99ba77 (stale) Mirror 11: 0x5c99ba77 (stale) Mirror 12: 0x5c99ba77 (stale) Mirror 13: 0x5c99ba77 (stale) Mirror 14: 0x5c99ba77 (stale) Mirror 15: 0x5c99ba77 (stale) Mirror 17: 0x5c99ba77 (stale) Mirror 18: 0x5c99ba77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cf0c): Mirror 9: 0x5c99ba77 (stale) Mirror 11: 0x5c99ba77 (stale) Mirror 12: 0x5c99ba77 (stale) Mirror 13: 0x5c99ba77 (stale) Mirror 14: 0x5c99ba77 (stale) Mirror 15: 0x5c99ba77 (stale) Mirror 17: 0x5c99ba77 (stale) Mirror 18: 0x5c99ba77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cf0c: skipped 000467 1786757671.114058 mapwrite 0x03e536 thru 0x03ffff (0x1aca bytes) 000468 1786757671.156973 trunc from 0x01bd68 to 0x028ce3 (0xcf7c bytes) 000469 1786757671.163765 read 0x01be51 thru 0x01c641 (0x7f1 bytes) 000470 1786757671.169061 punch from 0x0331d5 to 0x03ffff (0xce2b bytes) 000471 1786757671.173357 read 0x02b3e9 thru 0x02ca73 (0x168b bytes) 000472 1786757671.193582 mapwrite 0x01fbc4 thru 0x02ea4c (0xee89 bytes) 000473 1786757671.258258 trunc from 0x01ac30 to 0x028f13 (0xe2e4 bytes) 000474 1786757671.266291 write 0x03b14e thru 0x03b25f (0x112 bytes) 000475 1786757671.273795 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x73c92603 (stale) Mirror 11: 0x73c92603 (stale) Mirror 12: 0x73c92603 (stale) Mirror 13: 0x73c92603 (stale) Mirror 14: 0x73c92603 (stale) Mirror 15: 0x73c92603 (stale) Mirror 17: 0x73c92603 (stale) Mirror 18: 0x73c92603 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x417c535a Mirror 19: 0x417c535a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000476 1786757673.095032 fallocate 0x01e707 thru 0x022c0e (0x4508 bytes) 000478 1786757673.203172 fallocate 0x00090c thru 0x000fde (0x6d3 bytes) 000479 1786757673.212553 read 0x011011 thru 0x01f6ff (0xe6ef bytes) 000480 1786757673.235661 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x73c92603 (stale) Mirror 11: 0x73c92603 (stale) Mirror 12: 0x73c92603 (stale) Mirror 13: 0x73c92603 (stale) Mirror 14: 0x73c92603 (stale) Mirror 15: 0x73c92603 (stale) Mirror 17: 0x73c92603 (stale) Mirror 18: 0x73c92603 (stale) Mirror 19: 0x417c535a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x417c535a Mirror 20: 0x417c535a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000481 1786757674.740728 mapwrite 0x01cc45 thru 0x02bd93 (0xf14f bytes) 000482 1786757675.024858 mapwrite 0x0344ec thru 0x03649f (0x1fb4 bytes) 000483 1786757675.091708 trunc from 0x0298ba to 0x036b98 (0xd2df bytes) 000484 1786757675.107386 read 0x02c1bc thru 0x02f0f3 (0x2f38 bytes) 000485 1786757675.114371 trunc from 0x040000 to 0x010abd (0x2f542 bytes) 000486 1786757675.154584 trunc from 0x010abe to 0x022d26 (0x12269 bytes) 000487 1786757675.176207 mapread 0x004ae9 thru 0x0119fd (0xcf15 bytes) 000488 1786757675.199517 mapread 0x02254e thru 0x022d26 (0x7d9 bytes) 000489 1786757675.211880 trunc from 0x017de7 to 0x02190a (0x9b24 bytes) 000490 1786757675.222248 MIRROR_RESYNC 11 mirrors 000491 1786757677.855998 MIRROR_EXTEND to 12 mirrors 000492 1786757678.809372 mapread 0x013a25 thru 0x022650 (0xec2c bytes) 000493 1786757678.849762 read 0x01f6d0 thru 0x022d26 (0x3657 bytes) 000494 1786757678.850711 mapwrite 0x024290 thru 0x02d45a (0x91cb bytes) 000495 1786757679.040905 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2d45b): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d45b): Mirror 9: 0xc43c3862 (stale) Mirror 11: 0xc43c3862 (stale) Mirror 12: 0xc43c3862 (stale) Mirror 13: 0xc43c3862 (stale) Mirror 14: 0xc43c3862 (stale) Mirror 15: 0xc43c3862 (stale) Mirror 17: 0xc43c3862 (stale) Mirror 18: 0xc43c3862 (stale) Mirror 19: 0xc43c3862 (stale) Mirror 20: 0xc43c3862 (stale) Mirror 21: 0xc43c3862 (stale) CRC-32 checksum value for chunk [0, 0x2d45b): Mirror 1: 0x1ff0df8d Mirror 22: 0x1ff0df8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d45b: skipped 000496 1786757680.486559 mapwrite 0x0349ec thru 0x03b453 (0x6a68 bytes) 000497 1786757680.674889 read 0x02d345 thru 0x034ab3 (0x776f bytes) 000498 1786757680.683982 trunc from 0x0102fe to 0x0182f4 (0x7ff7 bytes) 000500 1786757680.691928 write 0x0096e0 thru 0x013298 (0x9bb9 bytes) 000501 1786757680.707042 trunc from 0x00bcf6 to 0x016b00 (0xae0b bytes) 000502 1786757680.733514 MIRROR_RESYNC 13 mirrors 000503 1786757685.233065 mapread 0x001458 thru 0x00d8a3 (0xc44c bytes) 000504 1786757685.258347 trunc from 0x03552f to 0x039507 (0x3fd9 bytes) 000505 1786757685.296616 trunc from 0x009f65 to 0x011912 (0x79ae bytes) 000506 1786757685.309349 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b454): Mirror 9: 0x54e77ee1 (stale) Mirror 11: 0x54e77ee1 (stale) Mirror 12: 0x54e77ee1 (stale) Mirror 13: 0x54e77ee1 (stale) Mirror 14: 0x54e77ee1 (stale) Mirror 15: 0x54e77ee1 (stale) Mirror 17: 0x54e77ee1 (stale) Mirror 18: 0x54e77ee1 (stale) Mirror 19: 0x54e77ee1 (stale) Mirror 20: 0x54e77ee1 (stale) Mirror 22: 0x54e77ee1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b454): Mirror 9: 0x54e77ee1 (stale) Mirror 11: 0x54e77ee1 (stale) Mirror 12: 0x54e77ee1 (stale) Mirror 13: 0x54e77ee1 (stale) Mirror 14: 0x54e77ee1 (stale) Mirror 15: 0x54e77ee1 (stale) Mirror 17: 0x54e77ee1 (stale) Mirror 18: 0x54e77ee1 (stale) Mirror 19: 0x54e77ee1 (stale) Mirror 20: 0x54e77ee1 (stale) Mirror 22: 0x54e77ee1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b454: skipped 000507 1786757686.328963 fallocate 0x005929 thru 0x00670f (0xde7 bytes) 000508 1786757686.334053 mapread 0x027077 thru 0x030adc (0x9a66 bytes) 000509 1786757686.370493 trunc from 0x01c6c7 to 0x02399e (0x72d8 bytes) 000510 1786757686.375942 fallocate 0x00a6a2 thru 0x016687 (0xbfe6 bytes) 000511 1786757686.385519 mapread 0x01785c thru 0x021a5a (0xa1ff bytes) 000512 1786757686.411046 mapread 0x01ee52 thru 0x026a2f (0x7bde bytes) 000513 1786757686.412016 write 0x03cdd3 thru 0x03dc26 (0xe54 bytes) 000514 1786757686.420078 write 0x01caea thru 0x01d2b7 (0x7ce bytes) 000515 1786757686.421147 read 0x033c22 thru 0x03dc26 (0xa005 bytes) 000516 1786757686.442220 trunc from 0x038ffe to 0x03c160 (0x3163 bytes) 000517 1786757686.465066 trunc from 0x004dab to 0x0118d0 (0xcb26 bytes) 000518 1786757686.477703 read 0x022036 thru 0x0255db (0x35a6 bytes) 000519 1786757686.478907 mapwrite 0x003ca4 thru 0x013394 (0xf6f1 bytes) 000520 1786757686.552776 trunc from 0x016911 to 0x01a283 (0x3973 bytes) 000521 1786757686.562053 trunc from 0x026bdf to 0x02af55 (0x4377 bytes) 000522 1786757686.581035 punch from 0x0180fd to 0x0197ab (0x16af bytes) 000523 1786757686.595615 trunc from 0x03dc27 to 0x0005a8 (0x3d67e bytes) 000524 1786757686.624968 write 0x024017 thru 0x0294e0 (0x54ca bytes) 000525 1786757686.646691 MIRROR_RESYNC 12 mirrors 000526 1786757688.850624 MIRROR_RESYNC 12 mirrors 000527 1786757689.522929 MIRROR_RESYNC 12 mirrors 000528 1786757690.319063 mapwrite 0x0374a7 thru 0x03ab17 (0x3671 bytes) 000529 1786757690.450674 trunc from 0x0017d6 to 0x0066e0 (0x4f0b bytes) 000530 1786757690.468542 fallocate 0x001e25 thru 0x00cecb (0xb0a7 bytes) 000531 1786757690.481633 fallocate 0x02da6c thru 0x03a752 (0xcce7 bytes) 000532 1786757690.489783 trunc from 0x018251 to 0x02367d (0xb42d bytes) 000533 1786757690.496928 fallocate 0x0008c6 thru 0x002b1a (0x2255 bytes) 000534 1786757690.503778 trunc from 0x03ab18 to 0x0229b5 (0x18162 bytes) 000535 1786757690.522021 mapread 0x002015 thru 0x00cc12 (0xabfe bytes) 000536 1786757690.534850 fallocate 0x0266e7 thru 0x02d21c (0x6b36 bytes) 000537 1786757690.551812 fallocate 0x02bb60 thru 0x03621b (0xa6bc bytes) 000538 1786757690.562104 write 0x03f5ce thru 0x03ffff (0xa32 bytes) 000539 1786757690.563919 mapread 0x031e0f thru 0x0378e1 (0x5ad3 bytes) 000540 1786757690.580769 write 0x035c69 thru 0x0361f5 (0x58d bytes) 000541 1786757690.582892 write 0x007384 thru 0x00ac14 (0x3891 bytes) 000542 1786757690.588983 punch from 0x00a74b to 0x017b44 (0xd3fa bytes) 000543 1786757690.606915 trunc from 0x040000 to 0x003282 (0x3cd7d bytes) 000544 1786757690.623616 trunc from 0x003283 to 0x0307ae (0x2d52c bytes) 000545 1786757690.666307 mapread 0x00cd77 thru 0x011325 (0x45af bytes) 000546 1786757690.685594 punch from 0x0031cf to 0x009d51 (0x6b83 bytes) 000547 1786757690.692228 write 0x00d88b thru 0x00fded (0x2563 bytes) 000549 1786757690.695092 trunc from 0x0307af to 0x0388ef (0x8141 bytes) 000550 1786757690.725651 trunc from 0x0388f0 to 0x01a802 (0x1e0ed bytes) 000551 1786757690.750919 fallocate 0x02db56 thru 0x037ec0 (0xa36b bytes) 000552 1786757690.761580 trunc from 0x01effb to 0x025247 (0x624d bytes) 000553 1786757690.785575 fallocate 0x011cb7 thru 0x0136d5 (0x1a1f bytes) 000554 1786757690.809190 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ec1): Mirror 9: 0x2bb09813 (stale) Mirror 12: 0x2bb09813 (stale) Mirror 13: 0x2bb09813 (stale) Mirror 14: 0x2bb09813 (stale) Mirror 15: 0x2bb09813 (stale) Mirror 17: 0x2bb09813 (stale) Mirror 18: 0x2bb09813 (stale) Mirror 19: 0x2bb09813 (stale) Mirror 20: 0x2bb09813 (stale) Mirror 22: 0x2bb09813 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ec1): Mirror 9: 0x2bb09813 (stale) Mirror 12: 0x2bb09813 (stale) Mirror 13: 0x2bb09813 (stale) Mirror 14: 0x2bb09813 (stale) Mirror 15: 0x2bb09813 (stale) Mirror 17: 0x2bb09813 (stale) Mirror 18: 0x2bb09813 (stale) Mirror 19: 0x2bb09813 (stale) Mirror 20: 0x2bb09813 (stale) Mirror 22: 0x2bb09813 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ec1: skipped 000555 1786757691.460946 punch from 0x012f01 to 0x021b76 (0xec76 bytes) 000558 1786757691.474897 trunc from 0x01cecc to 0x01eadd (0x1c12 bytes) 000559 1786757691.481691 punch from 0x00952b to 0x00f806 (0x62dc bytes) 000560 1786757691.488561 write 0x026b6e thru 0x0297a9 (0x2c3c bytes) 000561 1786757691.507221 trunc from 0x0092e8 to 0x00b997 (0x26b0 bytes) 000562 1786757691.518926 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ec1): Mirror 12: 0x2bb09813 (stale) Mirror 13: 0x2bb09813 (stale) Mirror 14: 0x2bb09813 (stale) Mirror 15: 0x2bb09813 (stale) Mirror 17: 0x2bb09813 (stale) Mirror 18: 0x2bb09813 (stale) Mirror 19: 0x2bb09813 (stale) Mirror 20: 0x2bb09813 (stale) Mirror 22: 0x2bb09813 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ec1): Mirror 12: 0x2bb09813 (stale) Mirror 13: 0x2bb09813 (stale) Mirror 14: 0x2bb09813 (stale) Mirror 15: 0x2bb09813 (stale) Mirror 17: 0x2bb09813 (stale) Mirror 18: 0x2bb09813 (stale) Mirror 19: 0x2bb09813 (stale) Mirror 20: 0x2bb09813 (stale) Mirror 22: 0x2bb09813 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ec1: skipped 000564 1786757692.337192 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ec1): Mirror 12: 0x2bb09813 (stale) Mirror 13: 0x2bb09813 (stale) Mirror 14: 0x2bb09813 (stale) Mirror 15: 0x2bb09813 (stale) Mirror 18: 0x2bb09813 (stale) Mirror 19: 0x2bb09813 (stale) Mirror 20: 0x2bb09813 (stale) Mirror 22: 0x2bb09813 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ec1): Mirror 12: 0x2bb09813 (stale) Mirror 13: 0x2bb09813 (stale) Mirror 14: 0x2bb09813 (stale) Mirror 15: 0x2bb09813 (stale) Mirror 18: 0x2bb09813 (stale) Mirror 19: 0x2bb09813 (stale) Mirror 20: 0x2bb09813 (stale) Mirror 22: 0x2bb09813 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ec1: skipped 000565 1786757693.009165 punch from 0x000913 to 0x004529 (0x3c17 bytes) 000566 1786757693.028956 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ec1): Mirror 12: 0x2bb09813 (stale) Mirror 13: 0x2bb09813 (stale) Mirror 14: 0x2bb09813 (stale) Mirror 15: 0x2bb09813 (stale) Mirror 18: 0x2bb09813 (stale) Mirror 19: 0x2bb09813 (stale) Mirror 20: 0x2bb09813 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ec1): Mirror 12: 0x2bb09813 (stale) Mirror 13: 0x2bb09813 (stale) Mirror 14: 0x2bb09813 (stale) Mirror 15: 0x2bb09813 (stale) Mirror 18: 0x2bb09813 (stale) Mirror 19: 0x2bb09813 (stale) Mirror 20: 0x2bb09813 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ec1: skipped 000567 1786757693.941136 fallocate 0x016fa6 thru 0x024d8b (0xdde6 bytes) 000568 1786757693.954907 mapread 0x002c7c thru 0x007022 (0x43a7 bytes) 000570 1786757693.985678 write 0x0139ce thru 0x019815 (0x5e48 bytes) 000571 1786757694.006476 MIRROR_RESYNC 8 mirrors 000572 1786757696.794309 fallocate 0x01a458 thru 0x01aa68 (0x611 bytes) 000573 1786757696.828085 mapwrite 0x004359 thru 0x00cd3c (0x89e4 bytes) 000574 1786757696.889228 MIRROR_RESYNC 8 mirrors 000575 1786757699.681928 read 0x007343 thru 0x00817a (0xe38 bytes) 000576 1786757699.695781 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000577 1786757700.342917 write 0x01767f thru 0x026eef (0xf871 bytes) 000578 1786757700.446172 fallocate 0x0133de thru 0x01c45f (0x9082 bytes) 000579 1786757700.468224 read 0x03456d thru 0x037ec0 (0x3954 bytes) 000580 1786757700.492865 punch from 0x004adc to 0x0078e4 (0x2e09 bytes) 000581 1786757700.514172 fallocate 0x03f78e thru 0x03ffff (0x872 bytes) 000582 1786757700.529890 trunc from 0x012034 to 0x01b4ca (0x9497 bytes) 000583 1786757700.574290 mapwrite 0x023f9f thru 0x02d659 (0x96bb bytes) 000584 1786757700.616366 trunc from 0x007482 to 0x016942 (0xf4c1 bytes) 000585 1786757700.622741 trunc from 0x00c6a4 to 0x011f10 (0x586d bytes) 000586 1786757700.630807 MIRROR_RESYNC 7 mirrors 000587 1786757703.296988 fallocate 0x002a0a thru 0x002fe0 (0x5d7 bytes) 000588 1786757703.362419 trunc from 0x037c50 to 0x037ec0 (0x271 bytes) 000589 1786757703.379663 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ec1): Mirror 12: 0xebc28f5d (stale) Mirror 13: 0xebc28f5d (stale) Mirror 15: 0xebc28f5d (stale) Mirror 19: 0xebc28f5d (stale) Mirror 20: 0xebc28f5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ec1): Mirror 12: 0xebc28f5d (stale) Mirror 13: 0xebc28f5d (stale) Mirror 15: 0xebc28f5d (stale) Mirror 19: 0xebc28f5d (stale) Mirror 20: 0xebc28f5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ec1: skipped 000590 1786757703.950939 trunc from 0x0062b6 to 0x00f2e9 (0x9034 bytes) 000591 1786757703.965622 mapwrite 0x03da45 thru 0x03e142 (0x6fe bytes) 000592 1786757704.046650 trunc from 0x03e143 to 0x0189e0 (0x25762 bytes) 000593 1786757704.095207 write 0x019996 thru 0x025dcd (0xc438 bytes) 000594 1786757704.098859 trunc from 0x025dce to 0x0265ca (0x7fd bytes) 000595 1786757704.137095 trunc from 0x0265cb to 0x033bc8 (0xd5fe bytes) 000597 1786757704.176521 mapwrite 0x00f3c4 thru 0x014537 (0x5174 bytes) 000598 1786757704.231514 MIRROR_RESYNC 6 mirrors 000599 1786757706.373615 mapread 0x022de0 thru 0x031e2e (0xf04f bytes) 000600 1786757706.404944 trunc from 0x033bc9 to 0x02c368 (0x7860 bytes) 000602 1786757706.464707 mapread 0x008d6d thru 0x015a36 (0xccca bytes) 000604 1786757706.487803 read 0x0037a0 thru 0x00d5f0 (0x9e51 bytes) 000605 1786757706.488776 trunc from 0x02c369 to 0x0178b8 (0x14ab0 bytes) 000606 1786757706.510629 mapread 0x001a89 thru 0x007712 (0x5c8a bytes) 000608 1786757706.512321 mapwrite 0x01dd98 thru 0x025621 (0x788a bytes) 000609 1786757706.559645 read 0x0221fa thru 0x025621 (0x3428 bytes) 000610 1786757706.560580 mapwrite 0x0313cc thru 0x03e640 (0xd275 bytes) 000611 1786757706.615370 read 0x00a5f0 thru 0x010c80 (0x6691 bytes) 000612 1786757706.616326 punch from 0x012495 to 0x022077 (0xfbe3 bytes) 000613 1786757706.628333 write 0x017978 thru 0x02309a (0xb723 bytes) 000614 1786757706.642884 read 0x03a220 thru 0x03e640 (0x4421 bytes) 000615 1786757706.644190 punch from 0x014184 to 0x017766 (0x35e3 bytes) 000616 1786757706.663199 MIRROR_RESYNC 6 mirrors 000617 1786757708.311302 MIRROR_EXTEND to 7 mirrors 000618 1786757709.103151 mapread 0x008bff thru 0x013835 (0xac37 bytes) 000619 1786757709.134249 punch from 0x01e0f0 to 0x029a27 (0xb938 bytes) 000620 1786757709.208318 write 0x02eec8 thru 0x0348e7 (0x5a20 bytes) 000622 1786757709.226058 trunc from 0x03e641 to 0x01fd56 (0x1e8ea bytes) 000624 1786757709.247168 mapwrite 0x019101 thru 0x019a0a (0x90a bytes) 000625 1786757709.286114 trunc from 0x01fd57 to 0x02cc5f (0xcf09 bytes) 000626 1786757709.321930 mapread 0x015158 thru 0x01ee4b (0x9cf4 bytes) 000627 1786757709.323186 read 0x0034ed thru 0x00c5ff (0x9113 bytes) 000629 1786757709.344942 mapread 0x006570 thru 0x0087b8 (0x2249 bytes) 000630 1786757709.350173 read 0x008a36 thru 0x011600 (0x8bcb bytes) 000631 1786757709.371305 trunc from 0x02606a to 0x02c914 (0x68ab bytes) 000632 1786757709.380919 mapread 0x019769 thru 0x01e001 (0x4899 bytes) 000633 1786757709.388680 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cc60): Mirror 13: 0x32445b73 (stale) Mirror 15: 0x32445b73 (stale) Mirror 19: 0x32445b73 (stale) Mirror 20: 0x32445b73 (stale) Mirror 21: 0x32445b73 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cc60): Mirror 13: 0x32445b73 (stale) Mirror 15: 0x32445b73 (stale) Mirror 19: 0x32445b73 (stale) Mirror 20: 0x32445b73 (stale) Mirror 21: 0x32445b73 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cc60: skipped 000634 1786757709.811659 write 0x02390f thru 0x0289c5 (0x50b7 bytes) 000635 1786757709.824865 mapread 0x01fdc8 thru 0x022652 (0x288b bytes) 000636 1786757709.841125 read 0x020518 thru 0x02cc5f (0xc748 bytes) 000637 1786757709.853576 MIRROR_RESYNC 6 mirrors 000638 1786757712.085708 trunc from 0x02cc60 to 0x009294 (0x239cb bytes) 000639 1786757712.131618 fallocate 0x01338f thru 0x01d0fd (0x9d6f bytes) 000640 1786757712.141534 mapwrite 0x038d44 thru 0x03f1c8 (0x6485 bytes) 000641 1786757712.204820 punch from 0x0042b8 to 0x010d6d (0xcab6 bytes) 000642 1786757712.210706 trunc from 0x03e962 to 0x03f1c8 (0x867 bytes) 000643 1786757712.215521 punch from 0x03d518 to 0x03f1c8 (0x1cb1 bytes) 000644 1786757712.221108 trunc from 0x00b653 to 0x00d7dc (0x218a bytes) 000645 1786757712.225871 mapread 0x031c30 thru 0x033c3c (0x200d bytes) 000646 1786757712.233896 trunc from 0x01785c to 0x01e4c6 (0x6c6b bytes) 000647 1786757712.238985 punch from 0x010d01 to 0x01a1e7 (0x94e7 bytes) 000648 1786757712.244199 trunc from 0x0208b4 to 0x02d755 (0xcea2 bytes) 000649 1786757712.249675 read 0x031d50 thru 0x03c02c (0xa2dd bytes) 000650 1786757712.250778 fallocate 0x00aa4c thru 0x00dca4 (0x3259 bytes) 000651 1786757712.255897 mapwrite 0x03de6b thru 0x03ffff (0x2195 bytes) 000652 1786757712.292297 trunc from 0x02d716 to 0x03d3ff (0xfcea bytes) 000653 1786757712.298839 read 0x00b19e thru 0x00c741 (0x15a4 bytes) 000654 1786757712.329734 write 0x00d3cf thru 0x01c336 (0xef68 bytes) 000655 1786757712.332774 mapwrite 0x014d4e thru 0x0191f2 (0x44a5 bytes) 000656 1786757712.351830 write 0x03d8f5 thru 0x03ffff (0x270b bytes) 000657 1786757712.353582 mapwrite 0x03075f thru 0x033cbc (0x355e bytes) 000658 1786757712.375887 mapwrite 0x0260c8 thru 0x02b239 (0x5172 bytes) 000659 1786757712.395250 fallocate 0x031415 thru 0x039847 (0x8433 bytes) 000660 1786757712.402007 mapwrite 0x03dec2 thru 0x03ffff (0x213e bytes) 000661 1786757712.419526 mapread 0x03a7ad thru 0x03f1ba (0x4a0e bytes) 000662 1786757712.422347 trunc from 0x03a299 to 0x03b7cb (0x1533 bytes) 000663 1786757712.427681 trunc from 0x009401 to 0x00aedf (0x1adf bytes) 000664 1786757712.432804 punch from 0x012309 to 0x019e48 (0x7b40 bytes) 000665 1786757712.438769 mapwrite 0x03ec55 thru 0x03ffff (0x13ab bytes) 000666 1786757712.454303 write 0x019277 thru 0x01b58e (0x2318 bytes) 000667 1786757712.460706 read 0x013059 thru 0x01678e (0x3736 bytes) 000668 1786757712.473614 write 0x03069e thru 0x033939 (0x329c bytes) 000669 1786757712.476094 mapread 0x03f785 thru 0x03ffff (0x87b bytes) 000670 1786757712.477359 write 0x039cb5 thru 0x03fa02 (0x5d4e bytes) 000672 1786757712.480808 write 0x03faef thru 0x03ffff (0x511 bytes) 000673 1786757712.481996 trunc from 0x040000 to 0x002bb1 (0x3d44e bytes) 000674 1786757712.504096 trunc from 0x001a98 to 0x002bb1 (0x111a bytes) 000675 1786757712.510872 write 0x031dda thru 0x0395c2 (0x77e9 bytes) 000676 1786757712.515946 write 0x0203ae thru 0x028f26 (0x8b79 bytes) 000677 1786757712.527159 trunc from 0x0395c3 to 0x000dbd (0x38805 bytes) 000678 1786757712.541629 trunc from 0x00099d to 0x000dbd (0x421 bytes) 000679 1786757712.549023 write 0x02c724 thru 0x034db2 (0x868f bytes) 000680 1786757712.550820 read 0x028889 thru 0x034db2 (0xc52a bytes) 000681 1786757712.560759 write 0x00af32 thru 0x017043 (0xc112 bytes) 000682 1786757712.563833 punch from 0x02e06d to 0x034db2 (0x6d46 bytes) 000683 1786757712.592730 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34db3): Mirror 13: 0xde165c8 (stale) Mirror 15: 0xde165c8 (stale) Mirror 19: 0xde165c8 (stale) Mirror 20: 0xde165c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34db3): Mirror 13: 0xde165c8 (stale) Mirror 15: 0xde165c8 (stale) Mirror 19: 0xde165c8 (stale) Mirror 20: 0xde165c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34db3: skipped 000684 1786757713.091292 punch from 0x0283ba to 0x032d46 (0xa98d bytes) 000685 1786757713.101724 write 0x00c032 thru 0x01b69c (0xf66b bytes) 000686 1786757713.136372 trunc from 0x0241d9 to 0x027dc9 (0x3bf1 bytes) 000687 1786757713.148349 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34db3): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34db3): Mirror 13: 0xde165c8 (stale) Mirror 15: 0xde165c8 (stale) Mirror 19: 0xde165c8 (stale) Mirror 20: 0xde165c8 (stale) CRC-32 checksum value for chunk [0, 0x34db3): Mirror 1: 0xb3f539d8 Mirror 21: 0xb3f539d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34db3: skipped 000689 1786757714.373279 write 0x03776b thru 0x03a584 (0x2e1a bytes) 000690 1786757714.418813 trunc from 0x03a585 to 0x0126ea (0x27e9a bytes) 000691 1786757714.440387 write 0x01fa46 thru 0x02b2b1 (0xb86c bytes) 000692 1786757714.446699 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b2b2): Mirror 13: 0xc512638d (stale) Mirror 15: 0xc512638d (stale) Mirror 19: 0xc512638d (stale) Mirror 21: 0x65ea472b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b2b2): Mirror 13: 0xc512638d (stale) Mirror 15: 0xc512638d (stale) Mirror 19: 0xc512638d (stale) Mirror 21: 0x65ea472b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b2b2: skipped 000693 1786757714.941037 trunc from 0x021c65 to 0x02a281 (0x861d bytes) 000694 1786757714.956923 read 0x01448f thru 0x01d218 (0x8d8a bytes) 000695 1786757714.967595 mapwrite 0x00cca1 thru 0x01792f (0xac8f bytes) 000697 1786757715.088099 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b2b2): Mirror 13: 0xc512638d (stale) Mirror 19: 0xc512638d (stale) Mirror 21: 0x65ea472b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b2b2): Mirror 13: 0xc512638d (stale) Mirror 19: 0xc512638d (stale) Mirror 21: 0x65ea472b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b2b2: skipped 000698 1786757715.615942 punch from 0x01c66a to 0x0220a7 (0x5a3e bytes) 000699 1786757715.626885 mapwrite 0x02528a thru 0x02b47f (0x61f6 bytes) 000700 1786757715.702821 mapread 0x000cdd thru 0x00969d (0x89c1 bytes) 000701 1786757715.715916 trunc from 0x005e9a to 0x011bc3 (0xbd2a bytes) 000702 1786757715.722996 trunc from 0x00eb9a to 0x010b70 (0x1fd7 bytes) 000703 1786757715.729109 fallocate 0x037ca2 thru 0x03c77b (0x4ada bytes) 000704 1786757715.741097 write 0x004c60 thru 0x007034 (0x23d5 bytes) 000705 1786757715.750785 trunc from 0x00de39 to 0x0167e3 (0x89ab bytes) 000707 1786757715.757804 mapread 0x015d17 thru 0x024f84 (0xf26e bytes) 000708 1786757715.801618 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2b480): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b480): Mirror 13: 0xfef8231e (stale) Mirror 19: 0xfef8231e (stale) Mirror 21: 0x5fa78a2f (stale) CRC-32 checksum value for chunk [0, 0x2b480): Mirror 1: 0x60dbba94 Mirror 22: 0x60dbba94 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b480: skipped 000709 1786757717.099718 write 0x015994 thru 0x01be5f (0x64cc bytes) 000710 1786757717.209792 trunc from 0x01a408 to 0x01b92e (0x1527 bytes) 000711 1786757717.238785 mapwrite 0x00d6aa thru 0x00e71b (0x1072 bytes) 000712 1786757717.266697 mapread 0x01a76a thru 0x0296c4 (0xef5b bytes) 000713 1786757717.294044 trunc from 0x01efbe to 0x028e36 (0x9e79 bytes) 000714 1786757717.301953 mapread 0x012b40 thru 0x01b26e (0x872f bytes) 000715 1786757717.315849 read 0x0037e8 thru 0x010fc0 (0xd7d9 bytes) 000716 1786757717.330477 MIRROR_RESYNC 5 mirrors 000717 1786757719.311062 mapwrite 0x024182 thru 0x02d022 (0x8ea1 bytes) 000718 1786757719.452214 punch from 0x02c90f to 0x02d022 (0x714 bytes) 000719 1786757719.460802 read 0x02a9d3 thru 0x02d022 (0x2650 bytes) 000720 1786757719.469795 mapwrite 0x023941 thru 0x031f9c (0xe65c bytes) 000721 1786757719.557510 trunc from 0x00a8ea to 0x018a5c (0xe173 bytes) 000722 1786757719.572932 MIRROR_RESYNC 5 mirrors 000723 1786757721.394234 MIRROR_RESYNC 5 mirrors 000724 1786757721.895944 mapread 0x011258 thru 0x0204e3 (0xf28c bytes) 000725 1786757721.960120 mapread 0x024b31 thru 0x031f9c (0xd46c bytes) 000726 1786757721.982077 fallocate 0x01727f thru 0x0213b5 (0xa137 bytes) 000727 1786757722.023069 mapwrite 0x02671b thru 0x031533 (0xae19 bytes) 000728 1786757722.087621 punch from 0x02abd6 to 0x02c7f5 (0x1c20 bytes) 000729 1786757722.098450 fallocate 0x009bb1 thru 0x016bb9 (0xd009 bytes) 000731 1786757722.104967 trunc from 0x031f9d to 0x02d4a0 (0x4afc bytes) 000732 1786757722.144009 mapread 0x00e8c7 thru 0x01a3e4 (0xbb1e bytes) 000734 1786757722.165572 punch from 0x000104 to 0x0096bb (0x95b8 bytes) 000735 1786757722.177952 read 0x012e41 thru 0x01ce8e (0xa04e bytes) 000736 1786757722.178722 trunc from 0x005880 to 0x00903b (0x37bc bytes) 000737 1786757722.188233 mapwrite 0x03d35d thru 0x03ffff (0x2ca3 bytes) 000739 1786757722.265058 fallocate 0x02f641 thru 0x037c68 (0x8628 bytes) 000740 1786757722.285335 write 0x036088 thru 0x03ffff (0x9f78 bytes) 000741 1786757722.286663 trunc from 0x040000 to 0x008979 (0x37686 bytes) 000742 1786757722.313323 mapread 0x002ffb thru 0x008979 (0x597f bytes) 000743 1786757722.338674 trunc from 0x0007ce to 0x008979 (0x81ac bytes) 000744 1786757722.353647 MIRROR_RESYNC 5 mirrors 000745 1786757723.242268 trunc from 0x00055f to 0x008979 (0x841b bytes) 000746 1786757723.290759 mapread 0x002c7b thru 0x008979 (0x5cff bytes) 000747 1786757723.318276 fallocate 0x025f6c thru 0x033706 (0xd79b bytes) 000748 1786757723.326365 read 0x022b5a thru 0x02dc92 (0xb139 bytes) 000749 1786757723.344715 MIRROR_RESYNC 5 mirrors 000750 1786757724.614247 read 0x003eeb thru 0x00c198 (0x82ae bytes) 000751 1786757724.627166 mapread 0x03014a thru 0x033706 (0x35bd bytes) 000753 1786757724.659224 fallocate 0x02e090 thru 0x03c757 (0xe6c8 bytes) 000754 1786757724.692245 trunc from 0x033707 to 0x0153cb (0x1e33b bytes) 000756 1786757724.721591 punch from 0x00a9a1 to 0x0150a8 (0xa708 bytes) 000757 1786757724.727675 write 0x018aaf thru 0x01a487 (0x19d9 bytes) 000758 1786757724.732935 trunc from 0x01a488 to 0x0132a6 (0x71e1 bytes) 000759 1786757724.753664 mapread 0x00d157 thru 0x0132a6 (0x6150 bytes) 000761 1786757724.766422 punch from 0x0035f4 to 0x00eac0 (0xb4cd bytes) 000762 1786757724.772817 trunc from 0x0132a7 to 0x02a7bb (0x17515 bytes) 000763 1786757724.790305 trunc from 0x02a7bc to 0x004ac6 (0x25cf5 bytes) 000764 1786757724.818047 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4ac7): Mirror 13: 0xdd61f046 (stale) Mirror 19: 0xdd61f046 (stale) Mirror 21: 0xdd61f046 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4ac7): Mirror 13: 0xdd61f046 (stale) Mirror 19: 0xdd61f046 (stale) Mirror 21: 0xdd61f046 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4ac7: skipped 000765 1786757725.264791 fallocate 0x014cf1 thru 0x022df0 (0xe100 bytes) 000766 1786757725.271950 read 0x00141d thru 0x004ac6 (0x36aa bytes) 000767 1786757725.291802 write 0x00cbc9 thru 0x00e0d6 (0x150e bytes) 000768 1786757725.320285 punch from 0x000b87 to 0x006632 (0x5aac bytes) 000769 1786757725.333934 mapread 0x00b0d2 thru 0x00e0d6 (0x3005 bytes) 000770 1786757725.342886 trunc from 0x00e0d7 to 0x03cd98 (0x2ecc2 bytes) 000771 1786757725.369823 punch from 0x01f5fa to 0x02ccd9 (0xd6e0 bytes) 000772 1786757725.377732 trunc from 0x03cd99 to 0x008ef3 (0x33ea5 bytes) 000773 1786757725.407316 MIRROR_RESYNC 4 mirrors 000774 1786757726.191192 read 0x006e9f thru 0x008ef3 (0x2055 bytes) 000775 1786757726.197335 write 0x02cfbc thru 0x038b87 (0xbbcc bytes) 000776 1786757726.231990 trunc from 0x038b88 to 0x02bacd (0xd0ba bytes) 000777 1786757726.292980 trunc from 0x02bace to 0x02649c (0x5631 bytes) 000778 1786757726.324221 punch from 0x00baa9 to 0x01934a (0xd8a2 bytes) 000780 1786757726.332314 read 0x013552 thru 0x016795 (0x3244 bytes) 000781 1786757726.345863 trunc from 0x0089f5 to 0x010d0e (0x831a bytes) 000782 1786757726.354855 mapwrite 0x00bd49 thru 0x0126f0 (0x69a8 bytes) 000783 1786757726.408569 punch from 0x0150d9 to 0x01692c (0x1854 bytes) 000785 1786757726.422776 mapread 0x00eb7b thru 0x01b1df (0xc665 bytes) 000786 1786757726.446504 MIRROR_RESYNC 4 mirrors 000787 1786757727.756508 trunc from 0x004e86 to 0x00e280 (0x93fb bytes) 000788 1786757727.834002 trunc from 0x014c00 to 0x01c973 (0x7d74 bytes) 000789 1786757727.845718 punch from 0x013a38 to 0x021192 (0xd75b bytes) 000790 1786757727.859301 read 0x002b27 thru 0x00fe03 (0xd2dd bytes) 000791 1786757727.891024 trunc from 0x02649d to 0x00d37f (0x1911d bytes) 000792 1786757727.931779 trunc from 0x00d380 to 0x030b17 (0x23798 bytes) 000793 1786757727.979296 trunc from 0x030b18 to 0x0306c9 (0x44e bytes) 000794 1786757728.005836 trunc from 0x0306ca to 0x038d9b (0x86d2 bytes) 000795 1786757728.031354 trunc from 0x038d9c to 0x03287d (0x651e bytes) 000796 1786757728.055175 punch from 0x026212 to 0x026a0e (0x7fd bytes) 000797 1786757728.061146 write 0x01df84 thru 0x026560 (0x85dd bytes) 000798 1786757728.078783 punch from 0x029877 to 0x02f69f (0x5e29 bytes) 000800 1786757728.091893 trunc from 0x03287e to 0x0236e7 (0xf196 bytes) 000801 1786757728.122665 mapread 0x005b51 thru 0x011fae (0xc45e bytes) 000802 1786757728.157206 read 0x022865 thru 0x0236e7 (0xe83 bytes) 000803 1786757728.158630 punch from 0x016ff7 to 0x0200bc (0x90c6 bytes) 000804 1786757728.196896 write 0x02041d thru 0x027dcf (0x79b3 bytes) 000805 1786757728.208460 trunc from 0x027dd0 to 0x008072 (0x1fd5d bytes) 000806 1786757728.229653 mapwrite 0x00ef36 thru 0x0105ae (0x1679 bytes) 000807 1786757728.284362 fallocate 0x0356e7 thru 0x03ffff (0xa919 bytes) 000809 1786757728.295861 punch from 0x003286 to 0x006351 (0x30cc bytes) 000810 1786757728.304146 mapread 0x03dc14 thru 0x03ffff (0x23ec bytes) 000811 1786757728.330321 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x8f733af7 (stale) Mirror 19: 0x8f733af7 (stale) Mirror 21: 0x8f733af7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x83ecfe1 Mirror 22: 0x83ecfe1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000812 1786757729.280462 trunc from 0x038f80 to 0x03f64f (0x66d0 bytes) 000813 1786757729.331315 trunc from 0x040000 to 0x01e841 (0x217be bytes) 000814 1786757729.401855 write 0x028226 thru 0x02cb44 (0x491f bytes) 000815 1786757729.406875 mapread 0x028309 thru 0x02a29d (0x1f95 bytes) 000816 1786757729.413871 read 0x017f9b thru 0x01b914 (0x397a bytes) 000817 1786757729.424869 read 0x029e24 thru 0x02cb44 (0x2d21 bytes) 000819 1786757729.427789 punch from 0x012efc to 0x01fb69 (0xcc6e bytes) 000820 1786757729.441555 trunc from 0x01158f to 0x014f21 (0x3993 bytes) 000821 1786757729.452276 trunc from 0x014701 to 0x0187f7 (0x40f7 bytes) 000822 1786757729.463346 punch from 0x00c193 to 0x012244 (0x60b2 bytes) 000823 1786757729.473025 mapread 0x00c432 thru 0x013d3f (0x790e bytes) 000825 1786757729.518785 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cb45): Mirror 13: 0xf6f2bdda (stale) Mirror 19: 0xf6f2bdda (stale) Mirror 22: 0x9f645362 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cb45): Mirror 13: 0xf6f2bdda (stale) Mirror 19: 0xf6f2bdda (stale) Mirror 22: 0x9f645362 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cb45: skipped 000826 1786757730.109565 read 0x01ad2d thru 0x01db2c (0x2e00 bytes) 000827 1786757730.130808 trunc from 0x0263cb to 0x02cb44 (0x677a bytes) 000828 1786757730.145666 trunc from 0x02cb45 to 0x0161f1 (0x16953 bytes) 000829 1786757730.208600 write 0x00031a thru 0x00ee06 (0xeaed bytes) 000830 1786757730.232938 punch from 0x0051a8 to 0x006050 (0xea9 bytes) 000831 1786757730.259880 mapread 0x0109ed thru 0x0161f1 (0x5805 bytes) 000832 1786757730.327629 read 0x0020cf thru 0x011522 (0xf454 bytes) 000833 1786757730.335024 punch from 0x002154 to 0x010f10 (0xedbd bytes) 000834 1786757730.341886 read 0x0141fd thru 0x0161f1 (0x1ff5 bytes) 000835 1786757730.343135 mapwrite 0x03a936 thru 0x03bac5 (0x1190 bytes) 000836 1786757730.388605 mapwrite 0x031773 thru 0x03641e (0x4cac bytes) 000837 1786757730.427073 mapwrite 0x017be8 thru 0x01cff5 (0x540e bytes) 000838 1786757730.479596 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bac6): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bac6): Mirror 13: 0x5e75cd6c (stale) Mirror 19: 0x5e75cd6c (stale) Mirror 22: 0x4867fc61 (stale) CRC-32 checksum value for chunk [0, 0x3bac6): Mirror 1: 0x8d323715 Mirror 23: 0x8d323715 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bac6: skipped 000839 1786757731.984462 mapread 0x028be7 thru 0x02a6b5 (0x1acf bytes) 000840 1786757732.055653 trunc from 0x0132cf to 0x01bf6b (0x8c9d bytes) 000841 1786757732.099845 trunc from 0x03bac6 to 0x032179 (0x994c bytes) 000842 1786757732.124648 fallocate 0x025efd thru 0x02a6ea (0x47ee bytes) 000843 1786757732.130546 trunc from 0x0036bb to 0x0132ad (0xfbf3 bytes) 000845 1786757732.136267 trunc from 0x019760 to 0x01a4cc (0xd6d bytes) 000846 1786757732.142552 write 0x023b57 thru 0x0317eb (0xdc95 bytes) 000848 1786757732.155237 punch from 0x00bbc2 to 0x01260b (0x6a4a bytes) 000849 1786757732.163750 trunc from 0x03217a to 0x036d23 (0x4baa bytes) 000850 1786757732.184181 punch from 0x016748 to 0x01bce0 (0x5599 bytes) 000851 1786757732.197278 write 0x022681 thru 0x024795 (0x2115 bytes) 000853 1786757732.208203 read 0x034424 thru 0x036cbe (0x289b bytes) 000854 1786757732.215298 fallocate 0x01a013 thru 0x021af1 (0x7adf bytes) 000855 1786757732.221989 mapread 0x0011f5 thru 0x00c11d (0xaf29 bytes) 000856 1786757732.239810 mapread 0x01b15f thru 0x02a0a2 (0xef44 bytes) 000857 1786757732.258666 mapread 0x00d9d2 thru 0x0133cc (0x59fb bytes) 000858 1786757732.277250 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36d24): Mirror 13: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d24): Mirror 13: 0x5da101f5 (stale) Mirror 19: 0x5da101f5 (stale) Mirror 22: 0xabbe47d6 (stale) Mirror 23: 0x8e23c6ef (stale) CRC-32 checksum value for chunk [0, 0x36d24): Mirror 1: 0x246f9d71 Mirror 24: 0x246f9d71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d24: skipped 000859 1786757733.391143 read 0x02a21a thru 0x02a514 (0x2fb bytes) 000860 1786757733.417386 write 0x023718 thru 0x026d1c (0x3605 bytes) 000861 1786757733.522895 mapread 0x031747 thru 0x036d23 (0x55dd bytes) 000862 1786757733.565299 punch from 0x01a14d to 0x020b11 (0x69c5 bytes) 000863 1786757733.577834 fallocate 0x02de1e thru 0x0309be (0x2ba1 bytes) 000864 1786757733.586889 write 0x000f32 thru 0x009446 (0x8515 bytes) 000865 1786757733.617885 write 0x01e22e thru 0x028b67 (0xa93a bytes) 000866 1786757733.640049 read 0x00f00c thru 0x014143 (0x5138 bytes) 000867 1786757733.659138 fallocate 0x03ae98 thru 0x03ffff (0x5168 bytes) 000868 1786757733.678513 trunc from 0x036d24 to 0x018963 (0x1e3c0 bytes) 000869 1786757733.748510 mapwrite 0x02740f thru 0x030b1a (0x970c bytes) 000870 1786757733.968892 fallocate 0x032023 thru 0x034a11 (0x29ef bytes) 000871 1786757733.975445 mapwrite 0x00f3ec thru 0x01c17c (0xcd91 bytes) 000872 1786757734.057580 mapwrite 0x010b89 thru 0x017cd0 (0x7148 bytes) 000873 1786757734.085585 punch from 0x0001a1 to 0x0100ba (0xff1a bytes) 000874 1786757734.093841 mapwrite 0x02be24 thru 0x03b1da (0xf3b7 bytes) 000875 1786757734.191637 mapwrite 0x007e20 thru 0x00ae3b (0x301c bytes) 000876 1786757734.250927 trunc from 0x03b1db to 0x028430 (0x12daa bytes) 000877 1786757734.268688 mapwrite 0x039ffe thru 0x03ffff (0x6002 bytes) 000878 1786757734.331105 punch from 0x03b292 to 0x03ffff (0x4d6e bytes) 000879 1786757734.338365 fallocate 0x015f90 thru 0x01a494 (0x4505 bytes) 000880 1786757734.344648 write 0x01601c thru 0x01b07e (0x5063 bytes) 000881 1786757734.347339 trunc from 0x031ff1 to 0x03b46b (0x947b bytes) 000882 1786757734.353192 trunc from 0x00efee to 0x0130ef (0x4102 bytes) 000883 1786757734.361772 read 0x03186c thru 0x03ffff (0xe794 bytes) 000884 1786757734.377807 trunc from 0x040000 to 0x0025ad (0x3da52 bytes) 000886 1786757734.397925 mapread 0x0020d5 thru 0x0025ad (0x4d9 bytes) 000888 1786757734.401826 read 0x0017e9 thru 0x0025ad (0xdc5 bytes) 000889 1786757734.404564 mapwrite 0x03d6c0 thru 0x03ffff (0x2940 bytes) 000890 1786757734.506643 punch from 0x00a61f to 0x00ecf4 (0x46d6 bytes) 000891 1786757734.516755 trunc from 0x040000 to 0x0164a9 (0x29b56 bytes) 000892 1786757734.541866 punch from 0x00ee2c to 0x00f65b (0x830 bytes) 000893 1786757734.549997 trunc from 0x00a289 to 0x011c97 (0x7a0f bytes) 000894 1786757734.556543 trunc from 0x0164aa to 0x02597a (0xf4d1 bytes) 000895 1786757734.574665 trunc from 0x02597b to 0x011d48 (0x13c32 bytes) 000896 1786757734.594284 mapread 0x006e29 thru 0x011d48 (0xaf20 bytes) 000897 1786757734.673590 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11d49): Mirror 13: 0xd99bb79a (stale) Mirror 22: 0x43bf22f2 (stale) Mirror 23: 0xf9afb586 (stale) Mirror 24: 0xf9afb586 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d49): Mirror 13: 0xd99bb79a (stale) Mirror 22: 0x43bf22f2 (stale) Mirror 23: 0xf9afb586 (stale) Mirror 24: 0xf9afb586 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d49: skipped 000898 1786757735.249627 trunc from 0x0119a3 to 0x011d48 (0x3a6 bytes) 000899 1786757735.265062 trunc from 0x00f0ad to 0x011d48 (0x2c9c bytes) 000900 1786757735.278853 read 0x00486f thru 0x00e211 (0x99a3 bytes) 000901 1786757735.316273 mapread 0x003733 thru 0x009e49 (0x6717 bytes) 000902 1786757735.351900 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x11d49): Mirror 13: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d49): Mirror 13: 0xd99bb79a (stale) Mirror 22: 0x43bf22f2 (stale) Mirror 23: 0xf9afb586 (stale) Mirror 24: 0xf9afb586 (stale) CRC-32 checksum value for chunk [0, 0x11d49): Mirror 1: 0xeb49ba36 Mirror 25: 0xeb49ba36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d49: skipped 000903 1786757736.600922 read 0x009627 thru 0x011d48 (0x8722 bytes) 000904 1786757736.624592 write 0x03e44a thru 0x03ffff (0x1bb6 bytes) 000905 1786757736.681786 read 0x03bc3e thru 0x03ffff (0x43c2 bytes) 000906 1786757736.711358 MIRROR_RESYNC 6 mirrors 000907 1786757738.333068 punch from 0x02525d to 0x02dedb (0x8c7f bytes) 000908 1786757738.368813 trunc from 0x040000 to 0x006129 (0x39ed6 bytes) 000909 1786757738.393732 mapwrite 0x0388d7 thru 0x03ffff (0x7729 bytes) 000910 1786757738.458778 trunc from 0x01b81d to 0x02b3d4 (0xfbb8 bytes) 000911 1786757738.465948 mapwrite 0x0015fa thru 0x007203 (0x5c0a bytes) 000912 1786757738.496236 punch from 0x026503 to 0x031968 (0xb466 bytes) 000913 1786757738.508045 trunc from 0x040000 to 0x00d8ca (0x32735 bytes) 000914 1786757738.538111 fallocate 0x02676b thru 0x02cb69 (0x63ff bytes) 000915 1786757738.551085 mapread 0x01adf9 thru 0x02526b (0xa473 bytes) 000916 1786757738.582652 read 0x001b4b thru 0x00cabe (0xaf74 bytes) 000917 1786757738.586518 trunc from 0x01c714 to 0x02c0ff (0xf9ec bytes) 000918 1786757738.593225 punch from 0x0258e4 to 0x02c551 (0x6c6e bytes) 000919 1786757738.609150 mapread 0x0092fc thru 0x014add (0xb7e2 bytes) 000920 1786757738.624572 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2cb6a): Mirror 13: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cb6a): Mirror 13: 0x6b88318b (stale) Mirror 22: 0x6b88318b (stale) Mirror 23: 0x6b88318b (stale) Mirror 24: 0x6b88318b (stale) Mirror 25: 0x6b88318b (stale) CRC-32 checksum value for chunk [0, 0x2cb6a): Mirror 1: 0xf3e146ee Mirror 26: 0xf3e146ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cb6a: skipped 000921 1786757739.889474 fallocate 0x009589 thru 0x00e96c (0x53e4 bytes) 000922 1786757739.938983 trunc from 0x02304a to 0x02adab (0x7d62 bytes) 000923 1786757739.946892 trunc from 0x01e4d3 to 0x0237d1 (0x52ff bytes) 000924 1786757739.971314 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cb6a): Mirror 13: 0x6b88318b (stale) Mirror 23: 0x6b88318b (stale) Mirror 24: 0x6b88318b (stale) Mirror 25: 0x6b88318b (stale) Mirror 26: 0xf3e146ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cb6a): Mirror 13: 0x6b88318b (stale) Mirror 23: 0x6b88318b (stale) Mirror 24: 0x6b88318b (stale) Mirror 25: 0x6b88318b (stale) Mirror 26: 0xf3e146ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cb6a: skipped 000925 1786757740.632025 punch from 0x0220b8 to 0x022353 (0x29c bytes) 000926 1786757740.638934 trunc from 0x02bc0f to 0x02cb69 (0xf5b bytes) 000927 1786757740.647419 mapread 0x0161e2 thru 0x01ff9b (0x9dba bytes) 000928 1786757740.721590 punch from 0x001b91 to 0x00332d (0x179d bytes) 000930 1786757740.735900 read 0x015c84 thru 0x018c81 (0x2ffe bytes) 000931 1786757740.739662 trunc from 0x02cb6a to 0x000f15 (0x2bc54 bytes) 000932 1786757740.786704 mapwrite 0x001452 thru 0x00b9ec (0xa59b bytes) 000933 1786757740.858628 mapread 0x003afe thru 0x00b490 (0x7993 bytes) 000934 1786757740.864545 fallocate 0x01deda thru 0x02032e (0x2455 bytes) 000936 1786757740.893279 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2032f): Mirror 13: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2032f): Mirror 13: 0x22a4edcf (stale) Mirror 23: 0x22a4edcf (stale) Mirror 24: 0x22a4edcf (stale) Mirror 25: 0x22a4edcf (stale) Mirror 26: 0x5f55ccbb (stale) CRC-32 checksum value for chunk [0, 0x2032f): Mirror 1: 0xd58c3e58 Mirror 27: 0xd58c3e58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2032f: skipped 000938 1786757742.086241 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2032f): Mirror 13: 0x22a4edcf (stale) Mirror 23: 0x22a4edcf (stale) Mirror 24: 0x22a4edcf (stale) Mirror 26: 0x5f55ccbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2032f): Mirror 13: 0x22a4edcf (stale) Mirror 23: 0x22a4edcf (stale) Mirror 24: 0x22a4edcf (stale) Mirror 26: 0x5f55ccbb (stale) CRC-32 checksum value for chunk [0, 0x2032f): Mirror 1: 0xd58c3e58 Mirror 27: 0xd58c3e58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2032f: skipped 000940 1786757742.937695 fallocate 0x00bfd2 thru 0x01343a (0x7469 bytes) 000942 1786757742.990271 trunc from 0x02032f to 0x03d2aa (0x1cf7c bytes) 000943 1786757743.013699 read 0x00d31d thru 0x017245 (0x9f29 bytes) 000944 1786757743.025574 mapwrite 0x02b3cc thru 0x0372f4 (0xbf29 bytes) 000945 1786757743.098097 fallocate 0x02eeea thru 0x035a83 (0x6b9a bytes) 000946 1786757743.110106 mapread 0x035a54 thru 0x03d2aa (0x7857 bytes) 000947 1786757743.117275 read 0x002785 thru 0x004a76 (0x22f2 bytes) 000948 1786757743.157197 mapwrite 0x0093e8 thru 0x014203 (0xae1c bytes) 000949 1786757743.193067 mapwrite 0x01a0c7 thru 0x028bce (0xeb08 bytes) 000950 1786757743.237370 trunc from 0x03d2ab to 0x01b8de (0x219cc bytes) 000951 1786757743.269719 fallocate 0x03d47a thru 0x03ffff (0x2b86 bytes) 000952 1786757743.278497 mapread 0x002580 thru 0x008cad (0x672e bytes) 000953 1786757743.282129 trunc from 0x01b8df to 0x03907a (0x1d79c bytes) 000954 1786757743.298585 punch from 0x00be05 to 0x010625 (0x4821 bytes) 000955 1786757743.305023 read 0x02c08b thru 0x02fd87 (0x3cfd bytes) 000956 1786757743.314367 read 0x00e3c5 thru 0x012e35 (0x4a71 bytes) 000957 1786757743.315551 trunc from 0x015563 to 0x018272 (0x2d10 bytes) 000958 1786757743.328828 trunc from 0x00bc2c to 0x015641 (0x9a16 bytes) 000959 1786757743.339279 mapwrite 0x002b82 thru 0x0064d0 (0x394f bytes) 000960 1786757743.376946 mapwrite 0x03472e thru 0x03ffff (0xb8d2 bytes) 000961 1786757743.482197 trunc from 0x021fc6 to 0x02b8b6 (0x98f1 bytes) 000962 1786757743.495350 mapwrite 0x0114b2 thru 0x01b63f (0xa18e bytes) 000963 1786757743.532681 trunc from 0x040000 to 0x019ec2 (0x2613d bytes) 000965 1786757743.556689 fallocate 0x01939b thru 0x019f55 (0xbbb bytes) 000967 1786757743.569067 trunc from 0x019f56 to 0x0189c4 (0x1591 bytes) 000968 1786757743.608257 trunc from 0x009aeb to 0x00ff02 (0x6418 bytes) 000969 1786757743.622213 read 0x00c193 thru 0x00cfe4 (0xe52 bytes) 000970 1786757743.632278 punch from 0x016a7f to 0x0189c4 (0x1f46 bytes) 000971 1786757743.653321 punch from 0x012f36 to 0x0189c4 (0x5a8f bytes) 000972 1786757743.662202 trunc from 0x0189c5 to 0x015385 (0x363f bytes) 000973 1786757743.683561 punch from 0x003608 to 0x0132b3 (0xfcac bytes) 000974 1786757743.696791 mapwrite 0x02bc71 thru 0x037b97 (0xbf27 bytes) 000976 1786757743.761898 trunc from 0x037b98 to 0x007bf9 (0x2ff9e bytes) 000977 1786757743.790186 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x7bfa): Mirror 13: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7bfa): Mirror 13: 0x8064b1a1 (stale) Mirror 23: 0x8064b1a1 (stale) Mirror 24: 0x8064b1a1 (stale) Mirror 26: 0x5aa6efb5 (stale) Mirror 27: 0xf8df709b (stale) CRC-32 checksum value for chunk [0, 0x7bfa): Mirror 1: 0x664118f7 Mirror 28: 0x664118f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7bfa: skipped 000978 1786757745.217697 write 0x036030 thru 0x03a703 (0x46d4 bytes) 000979 1786757745.284920 mapwrite 0x034cf8 thru 0x03ab71 (0x5e7a bytes) 000980 1786757745.369479 write 0x01bb6d thru 0x028189 (0xc61d bytes) 000982 1786757745.388339 fallocate 0x000d38 thru 0x007e2e (0x70f7 bytes) 000983 1786757745.394742 mapread 0x01ecc2 thru 0x027966 (0x8ca5 bytes) 000984 1786757745.405456 read 0x012cd9 thru 0x018d56 (0x607e bytes) 000985 1786757745.419573 read 0x02e60e thru 0x03ab71 (0xc564 bytes) 000986 1786757745.448804 trunc from 0x01ec0c to 0x020275 (0x166a bytes) 000987 1786757745.475223 trunc from 0x03ab72 to 0x00391c (0x37255 bytes) 000988 1786757745.543299 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x391d): Mirror 13: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x391d): Mirror 13: 0xf15bde05 (stale) Mirror 23: 0xf15bde05 (stale) Mirror 24: 0xf15bde05 (stale) Mirror 26: 0x40de85de (stale) Mirror 27: 0xe2fb3060 (stale) Mirror 28: 0xf1f28c09 (stale) CRC-32 checksum value for chunk [0, 0x391d): Mirror 1: 0xf1f28c09 Mirror 29: 0xf1f28c09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x391d: skipped 000989 1786757746.734339 read 0x000f36 thru 0x00391c (0x29e7 bytes) 000990 1786757746.754905 read 0x000961 thru 0x00391c (0x2fbc bytes) 000991 1786757746.759568 punch from 0x002d87 to 0x00391c (0xb96 bytes) 000992 1786757746.804997 mapwrite 0x00fe24 thru 0x01281d (0x29fa bytes) 000993 1786757746.862878 mapwrite 0x01f3ba thru 0x028e08 (0x9a4f bytes) 000994 1786757746.935363 write 0x020f55 thru 0x02dd65 (0xce11 bytes) 000995 1786757746.940112 punch from 0x00242f to 0x00fedf (0xdab1 bytes) 000996 1786757746.950319 write 0x0230a3 thru 0x030286 (0xd1e4 bytes) 000997 1786757746.952973 trunc from 0x018053 to 0x027641 (0xf5ef bytes) 000998 1786757746.980370 read 0x006766 thru 0x007a3e (0x12d9 bytes) 000999 1786757747.013155 mapwrite 0x023386 thru 0x0301bd (0xce38 bytes) All operations completed A-OK! PASS 70a (142s) == sanity-flr test 71: check mirror extend parent fid ==== 21:35:53 (1786757753) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x5e:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 21:36:04 (1786757764) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x29f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.226336 s, 9.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x29f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157427 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2a0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x29f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2a0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x29f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 21:36:13 (1786757773) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7236352 .. Extending file size to 2169248 .. Extending file size to 7439072 .. Extending file size to 4952448 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5893888 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8555904 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8013568 .. Extending file size to 8236736 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8478848 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8929088 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9079456 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 61285 61286 61289 61291 61292 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) 192.168.203.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:37:41 (1786757861) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1085728 .. Extending file size to 7231360 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8943424 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7754080 .. Extending file size to 9201888 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8957888 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62733 62734 62735 62737 62740 done 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) 192.168.203.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:39:01 (1786757941) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1268896 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1269228 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18736 3588284 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24388 3582632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43124 7170916 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1268896 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2076 1269228 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18736 3588284 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24388 3582632 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43124 7170916 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181015 s, 11.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182846 s, 11.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6e2a4ad7 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:39:21 (1786757961) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0881717 s, 23.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 21:39:28 (1786757968) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x182:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b6:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b6:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b7:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b8:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:39:35 (1786757975) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0916139 s, 22.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x50:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:39:43 (1786757983) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106374 s, 19.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2b9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2b9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ba:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:39:52 (1786757992) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161039 s, 13.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2bb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2bb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2bc:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:40:25 (1786758025) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg356-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796596 s, 13.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:41:12 (1786758072) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg356-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2bc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150812 s, 13.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2bd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2bc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2bd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:42:04 (1786758124) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg356-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg356-server: Pool lustre.test_204f created Adding targets to pool oleg356-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg356-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.271162 s, 7.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2bd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2be:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg356-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg356-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg356-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:43:18 (1786758198) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:43:26 (1786758206) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:43:35 (1786758215) oleg356-server: Pool lustre.test_206 created Adding targets to pool oleg356-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg356-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg356-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg356-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg356-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:44:04 (1786758244) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11867 s, 8.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293138 s, 35.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6c15844e Mirror 3: 0x6c15844e PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:44:17 (1786758257) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:44:21 (1786758261) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:44:24 (1786758264) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0921111 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x8e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1150000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1150000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1150000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb10000 Read this (4 bytes): BBBB 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:45:03 (1786758303) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0266411 s, 307 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:45:11 (1786758311) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701706 s, 14.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1a3 lmm_fid: [0x200000405:0x1a3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1a3 lmm_fid: [0x200000405:0x1a3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611555 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-14 21:45:13.000000000 -0400 Modify: 2026-08-14 21:45:13.000000000 -0400 Change: 2026-08-14 21:45:14.000000000 -0400 Birth: 2026-08-14 21:45:12.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:45:19 (1786758319) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043226 s, 24.3 MB/s oleg356-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg356-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg356-server: O/280000401/d8/2c8: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:45:41 (1786758341) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.388058 s, 106 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0227983 s, 180 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 14 21:45 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a8:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 14 21:45 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:45:50 (1786758350) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:46:00 (1786758360) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1873 sec =========== 21:46:09 (1786758369) === sanity-flr: start cleanup 21:46:11 (1786758371) === === sanity-flr: finish cleanup 21:46:19 (1786758379) ===