-----============= acceptance-small: sanity-flr ============----- Mon Aug 17 13:29:16 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg222-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 13:29:38 (1786987778) === oleg222-client.virtnet: executing check_config_client /mnt/lustre oleg222-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg222-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa028033ab000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa028033ab000.idle_timeout=debug disable quota as required oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:30:36 (1786987836) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8250] == sanity-flr test 0a: lfs mirror create with -N option == 13:30:39 (1786987839) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:31:00 (1786987860) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:31:04 (1786987864) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:31:08 (1786987868) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:31:39 (1786987899) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (6s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:31:44 (1786987904) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 13:31:49 (1786987909) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa028033ab000.stats=clear osc.lustre-OST0001-osc-ffffa028033ab000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:31:59 (1786987919) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:32:11 (1786987931) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:32:20 (1786987940) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:32:24 (1786987944) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:32:32 (1786987952) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 13:32:42 (1786987962) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:32:50 (1786987970) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:33:00 (1786987980) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:33:10 (1786987990) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:33:23 (1786988003) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:33:31 (1786988011) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.64807 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112372 s, 9.3 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:33:43 (1786988023) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.735057 s, 28.5 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:33:53 (1786988033) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (14s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:34:07 (1786988047) Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa028033ab000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa028033ab000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa028033ab000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa028033ab000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (98s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:35:45 (1786988145) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.429292 s, 14.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa028033ab000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa028033ab000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:36:36 (1786988196) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa028033ab000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa028033ab000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa028033ab000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa028033ab000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:38:03 (1786988283) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.31634 s, 15.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 17 13:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.459s user 0m0.001s sys 0m0.215s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa028033ab000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa028033ab000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.024s user 0m0.000s sys 0m0.167s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa028033ab000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa028033ab000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.570s user 0m0.000s sys 0m0.181s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa028033ab000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa028033ab000.ost_server_uuid in FULL state after 0 sec PASS 33b (113s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:39:56 (1786988396) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:40:00 (1786988400) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:40:04 (1786988404) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 13:40:08 (1786988408) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 13:40:15 (1786988415) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.968172 s, 13.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:40:29 (1786988429) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080226 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607135 s, 17.3 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:40:40 (1786988440) PASS 36c (13s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:40:52 (1786988452) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000746308 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0014143 s, 724 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000800348 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00120144 s, 852 kB/s flv=10 olv=10 PASS 36d (15s) == sanity-flr test 37: mirror I/O API verification ======= 13:41:07 (1786988467) create test files with size 15745905 .. f6816dc9ee06f947897ca838c6ac197d - 7aed468db7cda8bbc966fa7fcc11da14 - bdc252983877583c8edd3249d6772e3b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 13:41:32 (1786988492) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:41:51 (1786988511) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:42:00 (1786988520) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119131 s, 8.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353007 s, 29.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (14s) == sanity-flr test 41: lfs mirror resync check =========== 13:42:14 (1786988534) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0597545 s, 35.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.294279 s, 14.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205787 s, 10.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 13:42:32 (1786988552) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 13:42:36 (1786988556) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:42:40 (1786988560) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676493 s, 15.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 13:42:49 (1786988569) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.196092 s, 16.0 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:43:00 (1786988580) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:43:14 (1786988594) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.668602 s, 15.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:43:29 (1786988609) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa02802fae800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.246869 s, 42.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa0280a704000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 13:43:46 (1786988626) error lfs split: create victim file failed: File exists PASS 44f (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:44:03 (1786988643) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:44:39 (1786988679) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:45:22 (1786988722) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 13:45:26 (1786988726) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.248014 s, 12.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.153888 s, 20.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x697ce204 (nosync,stale) Mirror 2: 0x697ce204 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3e87d7007ab22ef2138a2ee2ca20e70b - mirror 1 checksum: 9a98cdad77e7040e6fe92f10c5b6cf30 - mirror 2 checksum: 3e87d7007ab22ef2138a2ee2ca20e70b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T13:45:30-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-17T13:45:30-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xff:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3e87d7007ab22ef2138a2ee2ca20e70b - mirror 1 checksum: 9a98cdad77e7040e6fe92f10c5b6cf30 - mirror 2 checksum: 9a98cdad77e7040e6fe92f10c5b6cf30 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T13:45:30-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-17T13:45:30-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xff:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:45:37 (1786988737) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384397 s, 27.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:45:45 (1786988745) Failing mds1 on oleg222-server Stopping /mnt/lustre-mds1 (opts:) on oleg222-server 13:45:51 (1786988751) shut down facet: mds1 facet_host: oleg222-server facet_failover_host: oleg222-server Failover mds1 to oleg222-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 13:46:18 (1786988778) targets are mounted 13:46:18 (1786988778) facet_failover done oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x9:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412989 s, 25.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000405 lmm_object_id: 0x9 lmm_fid: [0x240000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1f:0x0] } PASS 50A (51s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:46:36 (1786988796) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0313717 s, 653 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0135238 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:46:46 (1786988806) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.040286 s, 508 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0131647 s, 1.6 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:46:56 (1786988816) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00936666 s, 1.7 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:47:04 (1786988824) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0426465 s, 480 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:47:11 (1786988831) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00471199 s, 869 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0319766 s, 128 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:47:18 (1786988838) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:47:44 (1786988864) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (34s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:48:17 (1786988897) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:48:41 (1786988921) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:49:04 (1786988944) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 13:49:12 (1786988952) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 58270 mirror split pid 58271 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58270 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58271 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 13:50:20 (1786989020) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 9023 000001 1786989023.682507 write 0x0347e2 thru 0x0369f1 (0x2210 bytes) 000002 1786989023.717616 trunc from 0x01b1e4 to 0x02ab14 (0xf931 bytes) zero_range to largest ever: 0x3390c 000003 1786989023.728811 trunc from 0x032819 to 0x03390b (0x10f3 bytes) 000004 1786989023.743600 MIRROR_EXTEND to 2 mirrors 000005 1786989024.805054 mapwrite 0x026009 thru 0x035c6a (0xfc62 bytes) 000006 1786989024.964801 trunc from 0x0369f2 to 0x018bf0 (0x1de01 bytes) 000008 1786989025.049058 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000010 1786989025.458212 read 0x01355b thru 0x018bf0 (0x5696 bytes) 000011 1786989025.495523 write 0x012d7c thru 0x015c39 (0x2ebe bytes) 000012 1786989025.517652 mapread 0x0174fc thru 0x018bf0 (0x16f5 bytes) 000013 1786989025.518991 trunc from 0x008797 to 0x012c75 (0xa4df bytes) 000014 1786989025.551331 trunc from 0x00cdb8 to 0x00d4aa (0x6f3 bytes) 000016 1786989025.563718 trunc from 0x00be2f to 0x015515 (0x96e7 bytes) fallocating to largest ever: 0x390f7 000017 1786989025.586518 fallocate 0x03524c thru 0x0390f6 (0x3eab bytes) 000018 1786989025.605478 trunc from 0x00059d to 0x008cb9 (0x871d bytes) 000019 1786989025.618217 MIRROR_EXTEND to 2 mirrors 000021 1786989026.430807 punch from 0x00bc36 to 0x018bf0 (0xcfbb bytes) 000022 1786989026.495036 write 0x0093b0 thru 0x016159 (0xcdaa bytes) 000023 1786989026.514022 mapread 0x0177bd thru 0x018bf0 (0x1434 bytes) 000024 1786989026.546568 write 0x02dd39 thru 0x030d92 (0x305a bytes) 000025 1786989026.548504 punch from 0x026bbe to 0x02733d (0x780 bytes) 000026 1786989026.555771 trunc from 0x030d93 to 0x0330e5 (0x2353 bytes) 000028 1786989026.590963 mapwrite 0x032759 thru 0x0335d1 (0xe79 bytes) truncating to largest ever: 0x39a3d 000029 1786989026.642731 trunc from 0x0335d2 to 0x039a3c (0x646b bytes) 000030 1786989026.661612 write 0x02d29b thru 0x0371cf (0x9f35 bytes) 000031 1786989026.668841 fallocate 0x0375d9 thru 0x03964d (0x2075 bytes) 000033 1786989026.678743 MIRROR_RESYNC 2 mirrors 000035 1786989027.271132 mapread 0x00f411 thru 0x0134d0 (0x40c0 bytes) 000036 1786989027.294874 write 0x024595 thru 0x029d8d (0x57f9 bytes) 000037 1786989027.343532 read 0x01834c thru 0x02801b (0xfcd0 bytes) 000038 1786989027.359637 fallocate 0x02b25e thru 0x02f437 (0x41da bytes) 000039 1786989027.365060 mapwrite 0x03d1e1 thru 0x03ffff (0x2e1f bytes) 000040 1786989027.416987 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000041 1786989027.561039 read 0x03659c thru 0x03b2b4 (0x4d19 bytes) 000042 1786989027.575876 punch from 0x003e85 to 0x0056bf (0x183b bytes) zero_range to largest ever: 0x3a269 000043 1786989027.580497 trunc from 0x02a719 to 0x03a268 (0xfb50 bytes) 000044 1786989027.585890 fallocate 0x01bdac thru 0x01faae (0x3d03 bytes) 000047 1786989027.593402 mapread 0x00c303 thru 0x01b5bd (0xf2bb bytes) 000048 1786989027.622000 trunc from 0x040000 to 0x0276ff (0x18900 bytes) 000049 1786989027.656068 trunc from 0x020af1 to 0x024c87 (0x4197 bytes) 000050 1786989027.661809 mapwrite 0x0344ee thru 0x034bdd (0x6f0 bytes) 000051 1786989027.700461 write 0x01c11a thru 0x02b044 (0xef2b bytes) 000052 1786989027.709668 write 0x026e3b thru 0x031ff4 (0xb1ba bytes) 000053 1786989027.711714 read 0x023fc2 thru 0x02ab6f (0x6bae bytes) 000054 1786989027.717893 MIRROR_EXTEND to 2 mirrors 000055 1786989028.420011 punch from 0x019f9c to 0x0232b8 (0x931d bytes) 000057 1786989028.468377 MIRROR_RESYNC 2 mirrors 000058 1786989029.700706 fallocate 0x018719 thru 0x01adcc (0x26b4 bytes) 000060 1786989029.735565 write 0x03ca2d thru 0x03ffff (0x35d3 bytes) 000061 1786989029.737713 trunc from 0x01a759 to 0x01ded2 (0x377a bytes) 000062 1786989029.741600 punch from 0x025541 to 0x032a8a (0xd54a bytes) 000063 1786989029.751664 read 0x00713e thru 0x007904 (0x7c7 bytes) 000064 1786989029.761938 read 0x000679 thru 0x00945d (0x8de5 bytes) 000065 1786989029.778050 mapwrite 0x00b1e5 thru 0x00e243 (0x305f bytes) 000066 1786989029.818047 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x31623c03 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x79e17447 Mirror 3: 0x79e17447 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000067 1786989031.405680 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x31623c03 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x79e17447 Mirror 3: 0x79e17447 Mirror 4: 0x79e17447 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000068 1786989032.477084 mapread 0x017834 thru 0x017a11 (0x1de bytes) 000069 1786989032.492829 mapwrite 0x02304b thru 0x028d57 (0x5d0d bytes) 000070 1786989032.682134 trunc from 0x0379dd to 0x039228 (0x184c bytes) 000071 1786989032.705151 read 0x02ed02 thru 0x03b53a (0xc839 bytes) 000072 1786989032.754832 trunc from 0x0237e7 to 0x02be21 (0x863b bytes) 000073 1786989032.795316 MIRROR_RESYNC 4 mirrors 000074 1786989034.986870 fallocate 0x021a45 thru 0x02376b (0x1d27 bytes) 000075 1786989035.089727 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x11ae3dde (stale) Mirror 4: 0x11ae3dde (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x11ae3dde (stale) Mirror 4: 0x11ae3dde (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000076 1786989035.616194 MIRROR_RESYNC 3 mirrors 000077 1786989036.989412 trunc from 0x040000 to 0x027ead (0x18152 bytes) 000078 1786989037.041730 fallocate 0x01f0ca thru 0x022ffd (0x3f34 bytes) 000080 1786989037.056746 fallocate 0x0207b7 thru 0x0233e9 (0x2c33 bytes) 000081 1786989037.067595 trunc from 0x005f07 to 0x00a80f (0x4909 bytes) 000082 1786989037.088231 write 0x028a96 thru 0x0311e7 (0x8752 bytes) 000083 1786989037.095777 write 0x01514d thru 0x01c742 (0x75f6 bytes) 000084 1786989037.127067 read 0x016d69 thru 0x01adc4 (0x405c bytes) 000085 1786989037.128893 fallocate 0x009663 thru 0x012f19 (0x98b7 bytes) 000086 1786989037.137659 read 0x02f161 thru 0x0311e7 (0x2087 bytes) 000087 1786989037.143581 mapread 0x006cc7 thru 0x012d0d (0xc047 bytes) 000088 1786989037.178068 trunc from 0x02a0d5 to 0x02b323 (0x124f bytes) fallocating to largest ever: 0x3d89a 000089 1786989037.206657 fallocate 0x032feb thru 0x03d899 (0xa8af bytes) 000090 1786989037.228990 read 0x01ed91 thru 0x02af3a (0xc1aa bytes) 000091 1786989037.238548 punch from 0x022e21 to 0x030d17 (0xdef7 bytes) 000093 1786989037.252572 read 0x0170a0 thru 0x023f43 (0xcea4 bytes) 000094 1786989037.272536 punch from 0x007e69 to 0x008afd (0xc95 bytes) 000097 1786989037.297145 read 0x01a38c thru 0x02867f (0xe2f4 bytes) 000099 1786989037.301511 write 0x021366 thru 0x0223ea (0x1085 bytes) 000100 1786989037.330892 MIRROR_RESYNC 3 mirrors 000101 1786989038.329345 mapread 0x0012cc thru 0x002d5f (0x1a94 bytes) 000102 1786989038.365527 fallocate 0x029df3 thru 0x037c05 (0xde13 bytes) 000104 1786989038.497497 trunc from 0x00ca28 to 0x00eb09 (0x20e2 bytes) 000105 1786989038.505012 mapwrite 0x020da4 thru 0x028bd0 (0x7e2d bytes) 000106 1786989038.639953 mapread 0x0109ee thru 0x010c0a (0x21d bytes) 000107 1786989038.667601 trunc from 0x0299de to 0x0311e7 (0x780a bytes) 000108 1786989038.676561 read 0x01bd97 thru 0x01f6ad (0x3917 bytes) 000109 1786989038.677546 punch from 0x018766 to 0x018fd8 (0x873 bytes) 000110 1786989038.692403 punch from 0x00989e to 0x017c5b (0xe3be bytes) 000111 1786989038.700415 write 0x0097a0 thru 0x00ff34 (0x6795 bytes) 000112 1786989038.722026 trunc from 0x00ee45 to 0x01a42a (0xb5e6 bytes) 000113 1786989038.767596 read 0x023fde thru 0x03092c (0xc94f bytes) 000114 1786989038.786533 read 0x022044 thru 0x02bbae (0x9b6b bytes) 000115 1786989038.788151 trunc from 0x02f1c6 to 0x02f6cc (0x507 bytes) 000116 1786989038.795801 mapread 0x0051f8 thru 0x00ae87 (0x5c90 bytes) 000117 1786989038.796961 mapread 0x0102d2 thru 0x01ed20 (0xea4f bytes) 000118 1786989038.799525 read 0x0146cc thru 0x021b94 (0xd4c9 bytes) 000119 1786989038.800803 write 0x0009b7 thru 0x010487 (0xfad1 bytes) 000120 1786989038.809280 fallocate 0x006009 thru 0x010254 (0xa24c bytes) 000121 1786989038.818361 punch from 0x0210b1 to 0x029023 (0x7f73 bytes) 000122 1786989038.826735 trunc from 0x018c16 to 0x01d588 (0x4973 bytes) 000123 1786989038.836248 punch from 0x02be08 to 0x02c41f (0x618 bytes) 000124 1786989038.843425 mapread 0x01ff26 thru 0x02911f (0x91fa bytes) 000125 1786989038.859513 write 0x00b904 thru 0x017e95 (0xc592 bytes) 000126 1786989038.863838 read 0x0102a5 thru 0x01c36c (0xc0c8 bytes) 000127 1786989038.899477 trunc from 0x0311e8 to 0x01b9c4 (0x15823 bytes) 000128 1786989038.950547 read 0x015551 thru 0x01b9c4 (0x6474 bytes) 000129 1786989038.951691 fallocate 0x00b6ef thru 0x015052 (0x9964 bytes) 000130 1786989038.959532 mapwrite 0x01ddcf thru 0x023a7d (0x5caf bytes) 000131 1786989039.042520 write 0x03bfb6 thru 0x03ffff (0x404a bytes) 000132 1786989039.045419 read 0x005d66 thru 0x012137 (0xc3d2 bytes) 000133 1786989039.050965 mapwrite 0x010667 thru 0x01f9b9 (0xf353 bytes) 000134 1786989039.099537 trunc from 0x01487d to 0x015d62 (0x14e6 bytes) 000135 1786989039.106575 mapread 0x015b72 thru 0x018cf2 (0x3181 bytes) 000136 1786989039.126242 MIRROR_RESYNC 3 mirrors 000137 1786989040.097044 mapread 0x0038bb thru 0x00e07e (0xa7c4 bytes) 000138 1786989040.133504 punch from 0x02aab9 to 0x0360ff (0xb647 bytes) 000139 1786989040.252674 mapread 0x03a91e thru 0x03db9f (0x3282 bytes) 000140 1786989040.287625 read 0x038258 thru 0x03ffff (0x7da8 bytes) 000141 1786989040.290245 mapwrite 0x003c5b thru 0x00ba49 (0x7def bytes) 000142 1786989040.352849 fallocate 0x010b6e thru 0x01e497 (0xd92a bytes) 000143 1786989040.387745 MIRROR_RESYNC 3 mirrors 000144 1786989041.514317 trunc from 0x01f777 to 0x026299 (0x6b23 bytes) 000145 1786989041.601491 mapread 0x030729 thru 0x0326c1 (0x1f99 bytes) 000146 1786989041.673829 mapread 0x02f918 thru 0x038b78 (0x9261 bytes) 000147 1786989041.675149 fallocate 0x007a24 thru 0x00b6ed (0x3cca bytes) zero_range to largest ever: 0x3deea 000149 1786989041.681024 trunc from 0x0363ff to 0x03dee9 (0x7aeb bytes) 000150 1786989041.686884 punch from 0x02847f to 0x02e3c0 (0x5f42 bytes) 000151 1786989041.690678 write 0x01e516 thru 0x020f11 (0x29fc bytes) 000152 1786989041.699229 read 0x01eaf6 thru 0x027a02 (0x8f0d bytes) 000153 1786989041.705625 mapread 0x016c6f thru 0x0265a5 (0xf937 bytes) 000154 1786989041.718359 mapwrite 0x01272b thru 0x016bce (0x44a4 bytes) 000155 1786989041.770799 fallocate 0x00369a thru 0x00cb1b (0x9482 bytes) 000156 1786989041.779216 write 0x021a60 thru 0x022030 (0x5d1 bytes) 000157 1786989041.780459 trunc from 0x0188f0 to 0x01b9c1 (0x30d2 bytes) 000158 1786989041.791324 mapread 0x02849e thru 0x034c63 (0xc7c6 bytes) 000159 1786989041.815814 trunc from 0x040000 to 0x00eae7 (0x31518 bytes) 000160 1786989041.894718 mapwrite 0x0390fa thru 0x03ffff (0x6f06 bytes) 000161 1786989041.987083 read 0x014add thru 0x021698 (0xcbbc bytes) 000162 1786989042.007831 fallocate 0x0116d5 thru 0x0128b3 (0x11df bytes) 000163 1786989042.015306 read 0x033d74 thru 0x034cf9 (0xf86 bytes) 000164 1786989042.021700 mapread 0x015f05 thru 0x01c8b1 (0x69ad bytes) 000165 1786989042.037178 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x160a8600 (stale) Mirror 4: 0x160a8600 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6de2a5e5 Mirror 5: 0x6de2a5e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000166 1786989043.045155 punch from 0x02ca5f to 0x039813 (0xcdb5 bytes) 000167 1786989043.082496 mapwrite 0x01f073 thru 0x0208ff (0x188d bytes) 000169 1786989043.178597 mapread 0x0204be thru 0x0228b6 (0x23f9 bytes) 000171 1786989043.197288 punch from 0x02418e to 0x02f1fa (0xb06d bytes) 000172 1786989043.203754 mapread 0x0339b0 thru 0x03a2d7 (0x6928 bytes) 000173 1786989043.219634 trunc from 0x017585 to 0x01c642 (0x50be bytes) 000174 1786989043.224859 trunc from 0x01ba7a to 0x01ce07 (0x138e bytes) 000175 1786989043.230612 mapwrite 0x01f825 thru 0x02f702 (0xfede bytes) 000176 1786989043.280481 punch from 0x029c83 to 0x0305d1 (0x694f bytes) fallocating to largest ever: 0x40000 000177 1786989043.288379 fallocate 0x039bea thru 0x03ffff (0x6416 bytes) 000178 1786989043.298512 read 0x032c39 thru 0x03cd2f (0xa0f7 bytes) 000179 1786989043.305477 trunc from 0x040000 to 0x007b09 (0x384f6 bytes) 000180 1786989043.358454 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7b0a): Mirror 3: 0xec0f2b23 (stale) Mirror 5: 0xec0f2b23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7b0a): Mirror 3: 0xec0f2b23 (stale) Mirror 5: 0xec0f2b23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7b0a: skipped 000181 1786989043.807977 trunc from 0x002bf2 to 0x00313f (0x54e bytes) 000182 1786989043.827983 trunc from 0x007b0a to 0x02baa4 (0x23f9b bytes) 000183 1786989043.863556 trunc from 0x01ac63 to 0x01bd2b (0x10c9 bytes) 000185 1786989043.879503 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2baa5): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2baa5): Mirror 3: 0x1dd75f9d (stale) Mirror 5: 0x67f1f24f (stale) CRC-32 checksum value for chunk [0, 0x2baa5): Mirror 1: 0xf2ccc67a Mirror 6: 0xf2ccc67a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2baa5: skipped 000186 1786989045.071229 write 0x00e771 thru 0x01a4e1 (0xbd71 bytes) 000188 1786989045.166882 punch from 0x01a0b6 to 0x028b09 (0xea54 bytes) 000189 1786989045.208283 fallocate 0x00b688 thru 0x0140dc (0x8a55 bytes) 000190 1786989045.227543 read 0x0070bd thru 0x010e0f (0x9d53 bytes) 000191 1786989045.246348 trunc from 0x020685 to 0x02720c (0x6b88 bytes) 000193 1786989045.253289 mapread 0x002863 thru 0x010745 (0xdee3 bytes) 000194 1786989045.283207 trunc from 0x02baa5 to 0x00c374 (0x1f730 bytes) 000195 1786989045.320824 mapwrite 0x03f465 thru 0x03ffff (0xb9b bytes) 000196 1786989045.381594 read 0x027e05 thru 0x034435 (0xc631 bytes) 000197 1786989045.399854 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x160a8600 (stale) Mirror 5: 0x6de2a5e5 (stale) Mirror 6: 0xc638e508 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3b812a7c Mirror 7: 0x3b812a7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000198 1786989046.508031 punch from 0x02cf5c to 0x03116c (0x4211 bytes) 000199 1786989046.605830 mapread 0x01c7c3 thru 0x029b24 (0xd362 bytes) 000200 1786989046.695903 write 0x030cfc thru 0x033934 (0x2c39 bytes) 000201 1786989046.729459 MIRROR_RESYNC 5 mirrors 000202 1786989048.587938 mapread 0x025bf5 thru 0x02c80b (0x6c17 bytes) 000203 1786989048.665562 read 0x03005d thru 0x03a0f5 (0xa099 bytes) 000204 1786989048.686461 mapread 0x008a44 thru 0x00af82 (0x253f bytes) 000205 1786989048.701608 fallocate 0x014d1a thru 0x0180da (0x33c1 bytes) 000206 1786989048.759556 mapread 0x00e8f4 thru 0x01851a (0x9c27 bytes) 000207 1786989048.795035 fallocate 0x03119c thru 0x032d5b (0x1bc0 bytes) 000208 1786989048.805805 punch from 0x026000 to 0x027f8f (0x1f90 bytes) 000209 1786989048.831873 trunc from 0x025349 to 0x02ed3a (0x99f2 bytes) 000210 1786989048.843311 mapread 0x00cb6b thru 0x012cf9 (0x618f bytes) 000211 1786989048.854535 trunc from 0x00c0d4 to 0x015798 (0x96c5 bytes) 000213 1786989048.881403 MIRROR_RESYNC 5 mirrors 000214 1786989050.479465 trunc from 0x011162 to 0x0171f7 (0x6096 bytes) 000215 1786989050.525412 mapwrite 0x0107b4 thru 0x011e94 (0x16e1 bytes) 000216 1786989050.549795 trunc from 0x02b521 to 0x031e44 (0x6924 bytes) 000218 1786989050.558553 mapread 0x00f6c3 thru 0x01b462 (0xbda0 bytes) 000219 1786989050.582223 trunc from 0x040000 to 0x014575 (0x2ba8a bytes) 000220 1786989050.610357 punch from 0x007a5c to 0x0120c2 (0xa667 bytes) 000221 1786989050.622657 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x14576): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14576): Mirror 3: 0x280c62c2 (stale) Mirror 5: 0x280c62c2 (stale) Mirror 6: 0x280c62c2 (stale) Mirror 7: 0x280c62c2 (stale) CRC-32 checksum value for chunk [0, 0x14576): Mirror 1: 0x5b89551d Mirror 8: 0x5b89551d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14576: skipped 000222 1786989052.099894 trunc from 0x014576 to 0x0131e6 (0x138f bytes) 000223 1786989052.201720 fallocate 0x00b38d thru 0x019c46 (0xe8ba bytes) 000224 1786989052.224831 trunc from 0x019c47 to 0x033b05 (0x19ebf bytes) 000225 1786989052.265417 mapwrite 0x02792b thru 0x0352ce (0xd9a4 bytes) 000227 1786989052.424695 trunc from 0x0352cf to 0x009b8a (0x2b744 bytes) 000228 1786989052.493033 trunc from 0x009b8b to 0x0333b2 (0x29828 bytes) 000229 1786989052.537261 read 0x01455d thru 0x01e036 (0x9ada bytes) 000230 1786989052.553614 trunc from 0x0333b3 to 0x02ce1b (0x6597 bytes) 000231 1786989052.590967 mapread 0x019e54 thru 0x026bc9 (0xcd76 bytes) 000233 1786989052.610178 write 0x024725 thru 0x02eb45 (0xa421 bytes) 000234 1786989052.616625 read 0x015955 thru 0x01892b (0x2fd7 bytes) 000236 1786989052.622997 trunc from 0x02eb46 to 0x02357c (0xb5c9 bytes) 000237 1786989052.651895 write 0x029026 thru 0x02997e (0x959 bytes) 000238 1786989052.654891 fallocate 0x03748f thru 0x03a055 (0x2bc7 bytes) 000239 1786989052.672527 write 0x02650e thru 0x02fca6 (0x9799 bytes) 000240 1786989052.699019 mapwrite 0x03472f thru 0x036b0d (0x23df bytes) 000241 1786989052.740832 trunc from 0x0261be to 0x02e04c (0x7e8f bytes) 000242 1786989052.753866 fallocate 0x00d78c thru 0x01a9bd (0xd232 bytes) 000243 1786989052.771053 write 0x00321f thru 0x00eda8 (0xbb8a bytes) 000244 1786989052.774557 trunc from 0x03a056 to 0x009acf (0x30586 bytes) 000245 1786989052.801840 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9ad0): Mirror 5: 0xa704cdb1 (stale) Mirror 6: 0xa704cdb1 (stale) Mirror 7: 0xa704cdb1 (stale) Mirror 8: 0x4a1a58b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9ad0): Mirror 5: 0xa704cdb1 (stale) Mirror 6: 0xa704cdb1 (stale) Mirror 7: 0xa704cdb1 (stale) Mirror 8: 0x4a1a58b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9ad0: skipped 000246 1786989053.298677 trunc from 0x00584a to 0x009acf (0x4286 bytes) 000247 1786989053.307806 punch from 0x006f2f to 0x008979 (0x1a4b bytes) 000248 1786989053.313820 mapread 0x008167 thru 0x008709 (0x5a3 bytes) 000249 1786989053.323373 mapwrite 0x027d81 thru 0x02d15d (0x53dd bytes) 000250 1786989053.390147 trunc from 0x02d15e to 0x01c558 (0x10c05 bytes) 000251 1786989053.409585 mapread 0x011b89 thru 0x01be20 (0xa298 bytes) 000252 1786989053.426898 mapwrite 0x010308 thru 0x016fbe (0x6cb7 bytes) 000253 1786989053.447660 punch from 0x003841 to 0x00a207 (0x69c7 bytes) 000255 1786989053.454317 punch from 0x01415b to 0x01c558 (0x83fe bytes) 000257 1786989053.465310 mapwrite 0x02e52a thru 0x039e8b (0xb962 bytes) 000258 1786989053.523374 read 0x029be7 thru 0x033caa (0xa0c4 bytes) 000259 1786989053.524896 fallocate 0x037486 thru 0x03a625 (0x31a0 bytes) 000260 1786989053.532106 write 0x01ae98 thru 0x01ee72 (0x3fdb bytes) 000261 1786989053.543667 mapread 0x02477c thru 0x031a12 (0xd297 bytes) 000262 1786989053.561586 mapwrite 0x031519 thru 0x03cecc (0xb9b4 bytes) 000263 1786989053.616846 trunc from 0x0344dc to 0x03533e (0xe63 bytes) 000264 1786989053.622795 write 0x00e1fc thru 0x012662 (0x4467 bytes) 000265 1786989053.624291 mapread 0x006740 thru 0x013216 (0xcad7 bytes) 000267 1786989053.635661 fallocate 0x018248 thru 0x026115 (0xdece bytes) 000269 1786989053.642798 mapread 0x039962 thru 0x03c034 (0x26d3 bytes) 000270 1786989053.649859 read 0x002980 thru 0x00439a (0x1a1b bytes) 000271 1786989053.656400 read 0x01a6c2 thru 0x025321 (0xac60 bytes) 000272 1786989053.662447 trunc from 0x03cecd to 0x03541e (0x7aae bytes) 000273 1786989053.699357 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3541f): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3541f): Mirror 5: 0x225e62ff (stale) Mirror 6: 0x225e62ff (stale) Mirror 7: 0x225e62ff (stale) Mirror 8: 0x33f65a2 (stale) CRC-32 checksum value for chunk [0, 0x3541f): Mirror 1: 0x90caf26d Mirror 9: 0x90caf26d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3541f: skipped 000274 1786989055.151335 read 0x0312f1 thru 0x03541e (0x412e bytes) 000275 1786989055.163482 mapread 0x02eb88 thru 0x03541e (0x6897 bytes) 000276 1786989055.195512 read 0x000bbf thru 0x002713 (0x1b55 bytes) 000277 1786989055.230748 MIRROR_RESYNC 6 mirrors 000278 1786989057.109564 punch from 0x022cd9 to 0x026333 (0x365b bytes) 000280 1786989057.155739 punch from 0x0044b3 to 0x00cbdb (0x8729 bytes) 000281 1786989057.161235 mapwrite 0x03b0c0 thru 0x03ffff (0x4f40 bytes) 000282 1786989057.230258 write 0x01beda thru 0x01bef2 (0x19 bytes) 000283 1786989057.244332 trunc from 0x040000 to 0x0363b2 (0x9c4d bytes) 000285 1786989057.262463 mapread 0x029168 thru 0x02c677 (0x3510 bytes) 000286 1786989057.275223 fallocate 0x00bdbb thru 0x018310 (0xc556 bytes) 000287 1786989057.279873 write 0x0127ab thru 0x01da02 (0xb258 bytes) 000289 1786989057.300809 read 0x00fe5a thru 0x013384 (0x352b bytes) 000290 1786989057.307412 fallocate 0x029abe thru 0x02ff1d (0x6460 bytes) 000291 1786989057.315445 trunc from 0x0363b3 to 0x027740 (0xec72 bytes) 000292 1786989057.331828 trunc from 0x021738 to 0x027740 (0x6009 bytes) 000294 1786989057.338444 trunc from 0x02616b to 0x027740 (0x15d6 bytes) 000295 1786989057.342775 mapread 0x0120cc thru 0x0137f1 (0x1726 bytes) 000296 1786989057.344451 trunc from 0x01980c to 0x0238f9 (0xa0ee bytes) 000298 1786989057.378543 trunc from 0x008af0 to 0x015144 (0xc655 bytes) 000299 1786989057.399698 punch from 0x0099a4 to 0x017170 (0xd7cd bytes) 000300 1786989057.418203 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27741): Mirror 5: 0x13987b05 (stale) Mirror 7: 0x13987b05 (stale) Mirror 8: 0x13987b05 (stale) Mirror 9: 0x13987b05 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27741): Mirror 5: 0x13987b05 (stale) Mirror 7: 0x13987b05 (stale) Mirror 8: 0x13987b05 (stale) Mirror 9: 0x13987b05 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27741: skipped 000301 1786989057.839535 mapread 0x005115 thru 0x013eb6 (0xeda2 bytes) 000302 1786989057.875449 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x27741): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27741): Mirror 5: 0x13987b05 (stale) Mirror 7: 0x13987b05 (stale) Mirror 8: 0x13987b05 (stale) Mirror 9: 0x13987b05 (stale) CRC-32 checksum value for chunk [0, 0x27741): Mirror 1: 0xef80585a Mirror 10: 0xef80585a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27741: skipped 000303 1786989059.100420 fallocate 0x002645 thru 0x003c69 (0x1625 bytes) 000304 1786989059.166453 trunc from 0x01721d to 0x01e34c (0x7130 bytes) 000305 1786989059.179087 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x27741): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27741): Mirror 5: 0x13987b05 (stale) Mirror 7: 0x13987b05 (stale) Mirror 8: 0x13987b05 (stale) Mirror 9: 0x13987b05 (stale) Mirror 10: 0xef80585a (stale) CRC-32 checksum value for chunk [0, 0x27741): Mirror 1: 0x898c1423 Mirror 11: 0x898c1423 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27741: skipped 000306 1786989060.519227 trunc from 0x0215b4 to 0x027740 (0x618d bytes) 000307 1786989060.575108 write 0x003efb thru 0x00ac91 (0x6d97 bytes) 000308 1786989060.596244 mapread 0x01a95e thru 0x023bcd (0x9270 bytes) 000309 1786989060.637234 MIRROR_RESYNC 7 mirrors 000310 1786989063.343796 fallocate 0x030167 thru 0x03cdd7 (0xcc71 bytes) 000311 1786989063.419782 punch from 0x0111f4 to 0x01a092 (0x8e9f bytes) 000312 1786989063.434549 write 0x023fb1 thru 0x031858 (0xd8a8 bytes) 000313 1786989063.452842 mapwrite 0x01fe8e thru 0x024889 (0x49fc bytes) 000314 1786989063.560323 fallocate 0x00330f thru 0x011d8f (0xea81 bytes) 000315 1786989063.586097 read 0x0168de thru 0x0227d9 (0xbefc bytes) 000316 1786989063.607602 punch from 0x02406f to 0x031858 (0xd7ea bytes) 000317 1786989063.625126 mapread 0x015348 thru 0x01d589 (0x8242 bytes) 000318 1786989063.646102 punch from 0x0175d1 to 0x022355 (0xad85 bytes) 000319 1786989063.660730 punch from 0x008d2d to 0x009141 (0x415 bytes) 000320 1786989063.671627 mapwrite 0x016c9e thru 0x0232c9 (0xc62c bytes) 000321 1786989063.759823 fallocate 0x0298a4 thru 0x02ad0a (0x1467 bytes) 000322 1786989063.781816 trunc from 0x031859 to 0x03294f (0x10f7 bytes) 000323 1786989063.814227 punch from 0x00e0fb to 0x00fcf7 (0x1bfd bytes) 000324 1786989063.830372 MIRROR_RESYNC 7 mirrors 000325 1786989065.797029 fallocate 0x01790c thru 0x02075e (0x8e53 bytes) 000326 1786989065.881709 read 0x017925 thru 0x026b3f (0xf21b bytes) 000327 1786989065.910479 write 0x01652b thru 0x01d47e (0x6f54 bytes) 000328 1786989065.923811 mapwrite 0x01b208 thru 0x0287ca (0xd5c3 bytes) 000329 1786989066.019710 mapread 0x02ce9d thru 0x03294f (0x5ab3 bytes) 000330 1786989066.042145 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32950): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32950): Mirror 5: 0xda2523d7 (stale) Mirror 7: 0xda2523d7 (stale) Mirror 8: 0xda2523d7 (stale) Mirror 9: 0xda2523d7 (stale) Mirror 10: 0xda2523d7 (stale) Mirror 11: 0xda2523d7 (stale) CRC-32 checksum value for chunk [0, 0x32950): Mirror 1: 0x38d5d4e2 Mirror 12: 0x38d5d4e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32950: skipped 000331 1786989067.829433 fallocate 0x025c6c thru 0x02db99 (0x7f2e bytes) 000332 1786989067.947891 mapwrite 0x013270 thru 0x015784 (0x2515 bytes) 000333 1786989068.123528 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32950): Mirror 5: 0xda2523d7 (stale) Mirror 7: 0xda2523d7 (stale) Mirror 8: 0xda2523d7 (stale) Mirror 9: 0xda2523d7 (stale) Mirror 11: 0xda2523d7 (stale) Mirror 12: 0x38d5d4e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32950): Mirror 5: 0xda2523d7 (stale) Mirror 7: 0xda2523d7 (stale) Mirror 8: 0xda2523d7 (stale) Mirror 9: 0xda2523d7 (stale) Mirror 11: 0xda2523d7 (stale) Mirror 12: 0x38d5d4e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32950: skipped 000334 1786989069.087243 mapread 0x02476e thru 0x0289a0 (0x4233 bytes) 000335 1786989069.128384 trunc from 0x032950 to 0x023291 (0xf6be bytes) 000336 1786989069.158593 fallocate 0x02c8f0 thru 0x032605 (0x5d16 bytes) 000337 1786989069.167370 read 0x008efc thru 0x00cc7f (0x3d84 bytes) 000338 1786989069.187355 write 0x013070 thru 0x01ce53 (0x9de4 bytes) 000339 1786989069.202271 mapread 0x00ab23 thru 0x00bb49 (0x1027 bytes) 000340 1786989069.203397 trunc from 0x016779 to 0x0190ad (0x2935 bytes) 000341 1786989069.233124 read 0x005507 thru 0x01266e (0xd168 bytes) 000342 1786989069.252775 fallocate 0x029b48 thru 0x02fc73 (0x612c bytes) 000343 1786989069.278551 punch from 0x01ddf9 to 0x023291 (0x5499 bytes) 000344 1786989069.289738 fallocate 0x03cea7 thru 0x03ffff (0x3159 bytes) 000345 1786989069.319112 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf3a69933 (stale) Mirror 7: 0xf3a69933 (stale) Mirror 8: 0xf3a69933 (stale) Mirror 9: 0xf3a69933 (stale) Mirror 12: 0x4b066875 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf3a69933 (stale) Mirror 7: 0xf3a69933 (stale) Mirror 8: 0xf3a69933 (stale) Mirror 9: 0xf3a69933 (stale) Mirror 12: 0x4b066875 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000346 1786989070.096403 punch from 0x002620 to 0x005c42 (0x3623 bytes) 000347 1786989070.107892 mapread 0x02a8eb thru 0x02bd96 (0x14ac bytes) 000348 1786989070.136465 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xf3a69933 (stale) Mirror 7: 0xf3a69933 (stale) Mirror 8: 0xf3a69933 (stale) Mirror 9: 0xf3a69933 (stale) Mirror 12: 0x4b066875 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb617aa66 Mirror 13: 0xb617aa66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000349 1786989071.908561 punch from 0x02b25f to 0x030ffe (0x5da0 bytes) 000350 1786989071.964823 mapwrite 0x010559 thru 0x010fa4 (0xa4c bytes) 000351 1786989071.983372 write 0x007625 thru 0x00ec85 (0x7661 bytes) 000352 1786989072.004749 MIRROR_RESYNC 7 mirrors 000354 1786989075.643801 trunc from 0x00118c to 0x00aa6d (0x98e2 bytes) 000355 1786989075.749229 write 0x03e20d thru 0x03ffff (0x1df3 bytes) 000356 1786989075.763804 punch from 0x01317c to 0x017aa1 (0x4926 bytes) 000357 1786989075.773182 trunc from 0x040000 to 0x02eef5 (0x1110a bytes) 000358 1786989075.808526 read 0x023179 thru 0x02c645 (0x94cd bytes) 000359 1786989075.821571 trunc from 0x02eef6 to 0x01ebbb (0x1033a bytes) 000360 1786989075.871534 mapwrite 0x0028e9 thru 0x00a5c2 (0x7cda bytes) 000361 1786989076.006217 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ebbc): Mirror 5: 0xf8fc85f5 (stale) Mirror 7: 0xf8fc85f5 (stale) Mirror 9: 0xf8fc85f5 (stale) Mirror 12: 0xf8fc85f5 (stale) Mirror 13: 0xf8fc85f5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ebbc): Mirror 5: 0xf8fc85f5 (stale) Mirror 7: 0xf8fc85f5 (stale) Mirror 9: 0xf8fc85f5 (stale) Mirror 12: 0xf8fc85f5 (stale) Mirror 13: 0xf8fc85f5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ebbc: skipped 000362 1786989076.936224 fallocate 0x03fcb2 thru 0x03ffff (0x34e bytes) 000363 1786989076.948474 fallocate 0x0320d9 thru 0x03512d (0x3055 bytes) 000364 1786989076.958529 mapwrite 0x0331c3 thru 0x0354e9 (0x2327 bytes) 000365 1786989077.107061 trunc from 0x032a85 to 0x0354e9 (0x2a65 bytes) 000366 1786989077.119083 write 0x03b897 thru 0x03ffff (0x4769 bytes) 000368 1786989077.143592 trunc from 0x02db18 to 0x030e7c (0x3365 bytes) 000369 1786989077.178210 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdc22bc8c (stale) Mirror 7: 0xdc22bc8c (stale) Mirror 9: 0xdc22bc8c (stale) Mirror 12: 0xdc22bc8c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdc22bc8c (stale) Mirror 7: 0xdc22bc8c (stale) Mirror 9: 0xdc22bc8c (stale) Mirror 12: 0xdc22bc8c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000370 1786989077.920580 trunc from 0x00d3b8 to 0x018f61 (0xbbaa bytes) 000371 1786989077.946896 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xdc22bc8c (stale) Mirror 7: 0xdc22bc8c (stale) Mirror 9: 0xdc22bc8c (stale) Mirror 12: 0xdc22bc8c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa262374 Mirror 13: 0xa262374 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000372 1786989079.862872 punch from 0x011997 to 0x012ff4 (0x165e bytes) 000373 1786989079.933991 trunc from 0x02a221 to 0x038b3e (0xe91e bytes) 000374 1786989079.947698 fallocate 0x01a1ca thru 0x0220c6 (0x7efd bytes) 000375 1786989079.996322 mapread 0x0014a2 thru 0x00853e (0x709d bytes) 000377 1786989080.065751 trunc from 0x0166b3 to 0x01fc12 (0x9560 bytes) 000379 1786989080.089344 trunc from 0x016d8d to 0x018aca (0x1d3e bytes) 000380 1786989080.099541 read 0x0169d7 thru 0x01be02 (0x542c bytes) 000381 1786989080.115857 trunc from 0x03be1e to 0x03ffff (0x41e2 bytes) 000383 1786989080.129316 mapwrite 0x03894f thru 0x03ffff (0x76b1 bytes) 000384 1786989080.223631 mapwrite 0x010f80 thru 0x01ab7c (0x9bfd bytes) 000385 1786989080.301376 mapread 0x0183b2 thru 0x01d723 (0x5372 bytes) 000386 1786989080.329702 mapwrite 0x018d79 thru 0x023306 (0xa58e bytes) 000387 1786989080.404017 punch from 0x0191fc to 0x01d378 (0x417d bytes) 000388 1786989080.421258 punch from 0x02d441 to 0x032c28 (0x57e8 bytes) 000389 1786989080.436713 fallocate 0x000778 thru 0x005479 (0x4d02 bytes) 000390 1786989080.451208 fallocate 0x02579e thru 0x029fb5 (0x4818 bytes) 000391 1786989080.462200 read 0x0279d3 thru 0x033f93 (0xc5c1 bytes) 000393 1786989080.502548 write 0x0085e8 thru 0x014180 (0xbb99 bytes) 000394 1786989080.514398 punch from 0x02b6e4 to 0x030b30 (0x544d bytes) 000396 1786989080.526590 write 0x03e53c thru 0x03ffff (0x1ac4 bytes) 000398 1786989080.545375 MIRROR_RESYNC 6 mirrors 000399 1786989082.978422 trunc from 0x006bc2 to 0x012298 (0xb6d7 bytes) 000401 1786989083.032555 mapread 0x00f739 thru 0x01241e (0x2ce6 bytes) 000402 1786989083.066970 mapread 0x034e47 thru 0x03e2d0 (0x948a bytes) 000403 1786989083.085889 fallocate 0x01b89f thru 0x021a34 (0x6196 bytes) 000404 1786989083.093030 trunc from 0x012ad1 to 0x01b505 (0x8a35 bytes) 000405 1786989083.106419 MIRROR_RESYNC 6 mirrors 000407 1786989085.362466 fallocate 0x00dd3a thru 0x01c3e6 (0xe6ad bytes) 000408 1786989085.475138 mapwrite 0x00cb5e thru 0x01b25f (0xe702 bytes) 000409 1786989085.577096 read 0x004e80 thru 0x006cd0 (0x1e51 bytes) 000410 1786989085.582607 mapwrite 0x03d3b4 thru 0x03ffff (0x2c4c bytes) 000411 1786989085.607479 mapread 0x01db6e thru 0x027b49 (0x9fdc bytes) 000413 1786989085.620526 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x53149195 (stale) Mirror 9: 0x53149195 (stale) Mirror 12: 0x53149195 (stale) Mirror 13: 0x53149195 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x53149195 (stale) Mirror 9: 0x53149195 (stale) Mirror 12: 0x53149195 (stale) Mirror 13: 0x53149195 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000414 1786989086.065538 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x53149195 (stale) Mirror 9: 0x53149195 (stale) Mirror 12: 0x53149195 (stale) Mirror 13: 0x53149195 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xae1ceda5 Mirror 14: 0xae1ceda5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000415 1786989087.298005 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x53149195 (stale) Mirror 12: 0x53149195 (stale) Mirror 13: 0x53149195 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x53149195 (stale) Mirror 12: 0x53149195 (stale) Mirror 13: 0x53149195 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xae1ceda5 Mirror 14: 0xae1ceda5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000416 1786989088.170833 read 0x0397f6 thru 0x03a521 (0xd2c bytes) 000418 1786989088.182804 punch from 0x03b352 to 0x03ffff (0x4cae bytes) 000419 1786989088.238577 read 0x03d833 thru 0x03ffff (0x27cd bytes) 000420 1786989088.249612 trunc from 0x040000 to 0x0062b1 (0x39d4e bytes) 000421 1786989088.272905 mapwrite 0x02e347 thru 0x0328d4 (0x458e bytes) 000422 1786989088.373348 fallocate 0x01a284 thru 0x01a572 (0x2ef bytes) 000423 1786989088.378364 mapread 0x029b65 thru 0x0328d4 (0x8d70 bytes) 000424 1786989088.400180 write 0x035e51 thru 0x038f5b (0x310b bytes) 000425 1786989088.402435 punch from 0x02f71b to 0x033ace (0x43b4 bytes) 000426 1786989088.407876 read 0x0200ff thru 0x02c298 (0xc19a bytes) 000427 1786989088.417152 write 0x03200b thru 0x03beb3 (0x9ea9 bytes) 000428 1786989088.418688 write 0x014316 thru 0x01607e (0x1d69 bytes) 000429 1786989088.420480 mapwrite 0x022ed8 thru 0x025135 (0x225e bytes) 000430 1786989088.435891 mapwrite 0x000a5a thru 0x00506d (0x4614 bytes) 000431 1786989088.453892 mapwrite 0x03f912 thru 0x03ffff (0x6ee bytes) 000432 1786989088.489482 read 0x02a963 thru 0x03a81f (0xfebd bytes) 000433 1786989088.490377 write 0x013f1c thru 0x01ec77 (0xad5c bytes) 000434 1786989088.493469 trunc from 0x040000 to 0x004fec (0x3b013 bytes) 000435 1786989088.510221 write 0x0130c3 thru 0x01d95d (0xa89b bytes) 000436 1786989088.512464 mapread 0x011ec6 thru 0x01d95d (0xba98 bytes) 000437 1786989088.519111 mapwrite 0x0302e2 thru 0x036655 (0x6374 bytes) 000438 1786989088.565526 MIRROR_RESYNC 5 mirrors 000439 1786989089.439765 write 0x015ca1 thru 0x021ff6 (0xc356 bytes) 000440 1786989089.499438 trunc from 0x00645d to 0x011951 (0xb4f5 bytes) 000441 1786989089.506379 punch from 0x027dd6 to 0x030f32 (0x915d bytes) 000442 1786989089.511569 trunc from 0x00ff58 to 0x01ecf2 (0xed9b bytes) 000443 1786989089.527213 mapread 0x004d0c thru 0x014b7c (0xfe71 bytes) 000444 1786989089.559158 trunc from 0x001c92 to 0x0025ef (0x95e bytes) 000445 1786989089.564611 write 0x02fe9c thru 0x03a558 (0xa6bd bytes) 000446 1786989089.572218 mapread 0x00aeb5 thru 0x014b32 (0x9c7e bytes) 000447 1786989089.573455 trunc from 0x01e7e6 to 0x023105 (0x4920 bytes) 000449 1786989089.581402 mapwrite 0x01d4bf thru 0x021e60 (0x49a2 bytes) 000450 1786989089.613810 trunc from 0x003a39 to 0x00fb89 (0xc151 bytes) 000451 1786989089.620330 punch from 0x02f0fd to 0x03a236 (0xb13a bytes) 000453 1786989089.627141 mapwrite 0x022553 thru 0x0226ec (0x19a bytes) 000454 1786989089.639586 fallocate 0x038095 thru 0x03830c (0x278 bytes) 000455 1786989089.645067 write 0x03283a thru 0x039f73 (0x773a bytes) 000456 1786989089.657695 trunc from 0x003d59 to 0x004a41 (0xce9 bytes) 000458 1786989089.668825 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a559): Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a559): Mirror 7: 0x5299712f (stale) Mirror 12: 0x5299712f (stale) Mirror 13: 0x5299712f (stale) Mirror 14: 0x5299712f (stale) CRC-32 checksum value for chunk [0, 0x3a559): Mirror 1: 0x7de05180 Mirror 15: 0x7de05180 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a559: skipped 000459 1786989091.007605 mapwrite 0x016c6c thru 0x025cc0 (0xf055 bytes) 000460 1786989091.188550 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a559): Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a559): Mirror 7: 0x5299712f (stale) Mirror 12: 0x5299712f (stale) Mirror 13: 0x5299712f (stale) Mirror 14: 0x5299712f (stale) Mirror 15: 0x7de05180 (stale) CRC-32 checksum value for chunk [0, 0x3a559): Mirror 1: 0x1c1fea36 Mirror 16: 0x1c1fea36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a559: skipped 000461 1786989092.246363 read 0x01df58 thru 0x023702 (0x57ab bytes) 000462 1786989092.262488 punch from 0x00cc9a to 0x018911 (0xbc78 bytes) 000463 1786989092.354086 MIRROR_RESYNC 7 mirrors 000464 1786989094.353885 trunc from 0x02d491 to 0x034c54 (0x77c4 bytes) 000465 1786989094.426510 mapwrite 0x028316 thru 0x031460 (0x914b bytes) 000466 1786989094.509441 punch from 0x02b5c6 to 0x03813c (0xcb77 bytes) 000468 1786989094.519635 punch from 0x024b19 to 0x028b89 (0x4071 bytes) 000469 1786989094.527600 punch from 0x01cc79 to 0x01db88 (0xf10 bytes) 000470 1786989094.534398 read 0x02ed57 thru 0x03a558 (0xb802 bytes) 000471 1786989094.547688 mapread 0x002a68 thru 0x012740 (0xfcd9 bytes) 000472 1786989094.580836 trunc from 0x00c963 to 0x012fb8 (0x6656 bytes) 000473 1786989094.587563 write 0x02e940 thru 0x02f4b2 (0xb73 bytes) 000474 1786989094.595192 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a559): Mirror 12: 0xcefd7ee3 (stale) Mirror 13: 0xcefd7ee3 (stale) Mirror 14: 0xcefd7ee3 (stale) Mirror 15: 0xcefd7ee3 (stale) Mirror 16: 0xcefd7ee3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a559): Mirror 12: 0xcefd7ee3 (stale) Mirror 13: 0xcefd7ee3 (stale) Mirror 14: 0xcefd7ee3 (stale) Mirror 15: 0xcefd7ee3 (stale) Mirror 16: 0xcefd7ee3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a559: skipped 000475 1786989095.050603 trunc from 0x029cf0 to 0x02b559 (0x186a bytes) 000476 1786989095.065392 mapwrite 0x03292e thru 0x033577 (0xc4a bytes) 000477 1786989095.085226 write 0x036fc7 thru 0x03f3b9 (0x83f3 bytes) 000478 1786989095.095256 trunc from 0x006086 to 0x007b00 (0x1a7b bytes) 000479 1786989095.099210 trunc from 0x01d5f0 to 0x020b8e (0x359f bytes) 000481 1786989095.104444 punch from 0x03bfe7 to 0x03f3b9 (0x33d3 bytes) 000482 1786989095.113117 trunc from 0x03f3ba to 0x003fb5 (0x3b404 bytes) 000483 1786989095.127202 MIRROR_RESYNC 6 mirrors 000484 1786989095.983026 mapwrite 0x002ea1 thru 0x005c53 (0x2db3 bytes) 000485 1786989096.090228 trunc from 0x005c54 to 0x008ff3 (0x33a0 bytes) 000486 1786989096.110410 mapread 0x008274 thru 0x008ff3 (0xd80 bytes) 000487 1786989096.121856 mapread 0x0041d4 thru 0x008ff3 (0x4e20 bytes) 000488 1786989096.128118 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8ff4): Mirror 12: 0x94b52786 (stale) Mirror 13: 0x94b52786 (stale) Mirror 15: 0x94b52786 (stale) Mirror 16: 0x94b52786 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8ff4): Mirror 12: 0x94b52786 (stale) Mirror 13: 0x94b52786 (stale) Mirror 15: 0x94b52786 (stale) Mirror 16: 0x94b52786 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8ff4: skipped 000489 1786989096.587165 mapwrite 0x008432 thru 0x00e6cb (0x629a bytes) 000490 1786989096.659822 mapread 0x005338 thru 0x005c0c (0x8d5 bytes) 000491 1786989096.662034 read 0x00e522 thru 0x00e6cb (0x1aa bytes) 000492 1786989096.662868 trunc from 0x00e6cc to 0x013e30 (0x5765 bytes) 000493 1786989096.684804 punch from 0x00772d to 0x00f782 (0x8056 bytes) 000494 1786989096.692263 mapread 0x0086b5 thru 0x00abbb (0x2507 bytes) 000495 1786989096.715039 read 0x00cd14 thru 0x013a9d (0x6d8a bytes) 000497 1786989096.728302 trunc from 0x00f8e4 to 0x013e30 (0x454d bytes) 000498 1786989096.735494 write 0x0019ef thru 0x00ca89 (0xb09b bytes) 000499 1786989096.739267 write 0x0124a6 thru 0x02153a (0xf095 bytes) 000500 1786989096.751581 mapwrite 0x004cb3 thru 0x00df81 (0x92cf bytes) 000501 1786989096.828191 trunc from 0x02153b to 0x02ce51 (0xb917 bytes) 000502 1786989096.864043 punch from 0x014bf8 to 0x01f6e7 (0xaaf0 bytes) 000503 1786989096.884813 mapread 0x00b500 thru 0x018cbb (0xd7bc bytes) 000504 1786989096.968744 trunc from 0x02ce52 to 0x014251 (0x18c00 bytes) 000505 1786989097.001622 read 0x013b61 thru 0x014251 (0x6f1 bytes) 000506 1786989097.002503 trunc from 0x014252 to 0x03b8fa (0x276a9 bytes) 000507 1786989097.023062 trunc from 0x008e39 to 0x012e86 (0xa04e bytes) 000508 1786989097.028855 mapwrite 0x02e082 thru 0x037bfe (0x9b7d bytes) 000509 1786989097.065051 write 0x0243b3 thru 0x02ace4 (0x6932 bytes) 000511 1786989097.076859 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b8fb): Mirror 13: 0xea54b262 (stale) Mirror 15: 0xea54b262 (stale) Mirror 16: 0xea54b262 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8fb): Mirror 13: 0xea54b262 (stale) Mirror 15: 0xea54b262 (stale) Mirror 16: 0xea54b262 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8fb: skipped 000512 1786989097.576310 write 0x038f92 thru 0x03947e (0x4ed bytes) 000513 1786989097.588100 punch from 0x0328e0 to 0x03377a (0xe9b bytes) 000514 1786989097.594087 write 0x00b4cf thru 0x00e191 (0x2cc3 bytes) 000515 1786989097.602935 trunc from 0x0075e6 to 0x00e94b (0x7366 bytes) 000516 1786989097.615796 read 0x03717a thru 0x03b8fa (0x4781 bytes) 000517 1786989097.624986 trunc from 0x01eb88 to 0x02669e (0x7b17 bytes) 000518 1786989097.631247 mapread 0x02495b thru 0x02a36b (0x5a11 bytes) 000519 1786989097.664168 mapread 0x02e953 thru 0x02ff8b (0x1639 bytes) 000520 1786989097.676009 mapread 0x000033 thru 0x00b3c6 (0xb394 bytes) 000521 1786989097.699073 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b8fb): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8fb): Mirror 13: 0xea54b262 (stale) Mirror 15: 0xea54b262 (stale) Mirror 16: 0xea54b262 (stale) CRC-32 checksum value for chunk [0, 0x3b8fb): Mirror 1: 0xfe4bddc9 Mirror 17: 0xfe4bddc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8fb: skipped 000524 1786989098.814492 fallocate 0x0068ba thru 0x00715e (0x8a5 bytes) 000525 1786989098.863970 write 0x034754 thru 0x035a3e (0x12eb bytes) 000526 1786989098.883741 MIRROR_RESYNC 5 mirrors 000527 1786989100.455255 write 0x02367e thru 0x02ce78 (0x97fb bytes) 000528 1786989100.512320 mapread 0x02f81d thru 0x035b06 (0x62ea bytes) 000529 1786989100.535233 trunc from 0x03b8fb to 0x01d012 (0x1e8e8 bytes) 000530 1786989100.560774 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d013): Mirror 13: 0x9b490eb7 (stale) Mirror 16: 0x9b490eb7 (stale) Mirror 17: 0x9b490eb7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d013): Mirror 13: 0x9b490eb7 (stale) Mirror 16: 0x9b490eb7 (stale) Mirror 17: 0x9b490eb7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d013: skipped 000531 1786989100.895828 mapwrite 0x012801 thru 0x01cc38 (0xa438 bytes) 000532 1786989100.940758 read 0x014078 thru 0x01d012 (0x8f9b bytes) 000533 1786989100.945859 fallocate 0x01286f thru 0x01a82c (0x7fbe bytes) 000534 1786989100.954123 fallocate 0x0161b9 thru 0x016331 (0x179 bytes) 000535 1786989100.960227 trunc from 0x01d013 to 0x013ca0 (0x9372 bytes) 000536 1786989100.982168 mapwrite 0x015d3a thru 0x022118 (0xc3df bytes) 000537 1786989101.046198 mapwrite 0x01e889 thru 0x023830 (0x4fa8 bytes) 000538 1786989101.094979 trunc from 0x023831 to 0x03ef51 (0x1b721 bytes) 000539 1786989101.116608 punch from 0x017462 to 0x026424 (0xefc3 bytes) 000540 1786989101.125006 fallocate 0x01600e thru 0x01d83c (0x782f bytes) 000541 1786989101.134124 fallocate 0x03cc69 thru 0x03ffff (0x3397 bytes) 000542 1786989101.144905 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ef52): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef52): Mirror 13: 0xaa946dfb (stale) Mirror 16: 0xaa946dfb (stale) Mirror 17: 0xaa946dfb (stale) CRC-32 checksum value for chunk [0, 0x3ef52): Mirror 1: 0x338d2d93 Mirror 18: 0x338d2d93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef52: skipped 000544 1786989102.225859 mapread 0x02e7db thru 0x02f4d9 (0xcff bytes) 000545 1786989102.237788 mapread 0x039467 thru 0x03ef51 (0x5aeb bytes) 000546 1786989102.251758 trunc from 0x01139b to 0x01b0d8 (0x9d3e bytes) 000547 1786989102.295719 mapread 0x018ae1 thru 0x028563 (0xfa83 bytes) 000548 1786989102.323078 write 0x009d67 thru 0x01160e (0x78a8 bytes) 000549 1786989102.335077 fallocate 0x00b384 thru 0x018b7b (0xd7f8 bytes) 000550 1786989102.345042 mapread 0x01abae thru 0x029716 (0xeb69 bytes) 000551 1786989102.352057 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ef52): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef52): Mirror 13: 0xaa946dfb (stale) Mirror 16: 0xaa946dfb (stale) Mirror 17: 0xaa946dfb (stale) Mirror 18: 0x338d2d93 (stale) CRC-32 checksum value for chunk [0, 0x3ef52): Mirror 1: 0xeaaa9862 Mirror 19: 0xeaaa9862 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef52: skipped 000552 1786989103.351046 mapwrite 0x03978f thru 0x03ffff (0x6871 bytes) 000553 1786989103.475466 trunc from 0x040000 to 0x003c0d (0x3c3f2 bytes) 000554 1786989103.513442 write 0x038e51 thru 0x03ffff (0x71af bytes) 000556 1786989103.516782 trunc from 0x024782 to 0x027f8f (0x380e bytes) 000557 1786989103.526710 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x5390cfa9 (stale) Mirror 17: 0x5390cfa9 (stale) Mirror 18: 0x2576a275 (stale) Mirror 19: 0xbad60ff2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x5390cfa9 (stale) Mirror 17: 0x5390cfa9 (stale) Mirror 18: 0x2576a275 (stale) Mirror 19: 0xbad60ff2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000558 1786989104.019693 mapwrite 0x00eaf0 thru 0x01a6d3 (0xbbe4 bytes) 000559 1786989104.156240 punch from 0x02590d to 0x0319a3 (0xc097 bytes) 000560 1786989104.169828 write 0x03eaed thru 0x03ffff (0x1513 bytes) 000561 1786989104.194008 write 0x00db47 thru 0x010e0a (0x32c4 bytes) 000562 1786989104.195182 punch from 0x03cda5 to 0x03ffff (0x325b bytes) 000563 1786989104.244163 mapread 0x03a3de thru 0x03ffff (0x5c22 bytes) 000564 1786989104.279178 write 0x03db16 thru 0x03ffff (0x24ea bytes) 000565 1786989104.280520 fallocate 0x016a17 thru 0x01c118 (0x5702 bytes) 000566 1786989104.303390 trunc from 0x036658 to 0x03ffff (0x99a8 bytes) 000567 1786989104.341114 fallocate 0x02d4b4 thru 0x02e16c (0xcb9 bytes) 000568 1786989104.348570 trunc from 0x028fc7 to 0x034f39 (0xbf73 bytes) 000569 1786989104.354358 read 0x004cf0 thru 0x007da8 (0x30b9 bytes) 000570 1786989104.366434 punch from 0x029061 to 0x037d93 (0xed33 bytes) 000571 1786989104.377083 write 0x01f182 thru 0x022840 (0x36bf bytes) 000572 1786989104.385500 mapwrite 0x007a21 thru 0x014a18 (0xcff8 bytes) 000573 1786989104.495788 trunc from 0x040000 to 0x01709e (0x28f61 bytes) 000574 1786989104.526223 trunc from 0x01709f to 0x00a8df (0xc7bf bytes) 000575 1786989104.555464 read 0x009515 thru 0x00a8df (0x13cb bytes) 000576 1786989104.556201 fallocate 0x025006 thru 0x030cb9 (0xbcb4 bytes) 000577 1786989104.565070 mapwrite 0x03ee06 thru 0x03ffff (0x11fa bytes) 000578 1786989104.657065 trunc from 0x040000 to 0x034664 (0xb99b bytes) 000580 1786989104.683480 trunc from 0x001805 to 0x0024a7 (0xca3 bytes) 000581 1786989104.689026 mapread 0x00f520 thru 0x0195d2 (0xa0b3 bytes) 000582 1786989104.691176 trunc from 0x02f0b2 to 0x034664 (0x55b3 bytes) 000583 1786989104.697177 mapread 0x00a12c thru 0x00fb69 (0x5a3e bytes) 000584 1786989104.698497 mapwrite 0x01fbaf thru 0x027ae4 (0x7f36 bytes) 000585 1786989104.733229 read 0x02e064 thru 0x034664 (0x6601 bytes) 000586 1786989104.744894 fallocate 0x00cc57 thru 0x019370 (0xc71a bytes) 000587 1786989104.758392 MIRROR_RESYNC 5 mirrors 000588 1786989106.580947 punch from 0x00845d to 0x00c96b (0x450f bytes) 000589 1786989106.682029 read 0x020e9d thru 0x0299a4 (0x8b08 bytes) 000590 1786989106.714871 write 0x018262 thru 0x022838 (0xa5d7 bytes) 000591 1786989106.725232 fallocate 0x017077 thru 0x02211b (0xb0a5 bytes) 000592 1786989106.751204 punch from 0x013b30 to 0x01e2a8 (0xa779 bytes) 000593 1786989106.797046 punch from 0x00ca8e to 0x01aa38 (0xdfab bytes) 000595 1786989106.815954 mapread 0x001ff1 thru 0x00b59f (0x95af bytes) 000597 1786989106.856051 write 0x03d289 thru 0x03ffff (0x2d77 bytes) 000599 1786989106.861632 write 0x0276a0 thru 0x03055f (0x8ec0 bytes) 000600 1786989106.872407 MIRROR_RESYNC 5 mirrors 000601 1786989108.649324 fallocate 0x03b072 thru 0x03cdab (0x1d3a bytes) 000602 1786989108.702049 punch from 0x024a3e to 0x033451 (0xea14 bytes) 000603 1786989108.709025 write 0x0382a4 thru 0x03fd0d (0x7a6a bytes) 000604 1786989108.720996 write 0x006900 thru 0x00db9a (0x729b bytes) 000605 1786989108.731325 mapread 0x00bda5 thru 0x010cd3 (0x4f2f bytes) 000606 1786989108.759341 mapwrite 0x01c581 thru 0x01c930 (0x3b0 bytes) 000607 1786989108.817335 write 0x022955 thru 0x02467f (0x1d2b bytes) 000608 1786989108.818533 mapread 0x033665 thru 0x03a6d5 (0x7071 bytes) 000609 1786989108.842982 trunc from 0x0384ab to 0x03ffff (0x7b55 bytes) 000610 1786989108.853217 trunc from 0x00f949 to 0x0178da (0x7f92 bytes) 000611 1786989108.862772 fallocate 0x025d1e thru 0x02d280 (0x7563 bytes) 000612 1786989108.874281 trunc from 0x003f43 to 0x011ea4 (0xdf62 bytes) 000613 1786989108.885075 punch from 0x0120cf to 0x016d94 (0x4cc6 bytes) 000614 1786989108.894396 read 0x02ddf1 thru 0x03569e (0x78ae bytes) 000616 1786989108.897813 read 0x005a20 thru 0x00baad (0x608e bytes) 000617 1786989108.915844 trunc from 0x040000 to 0x01fa0d (0x205f2 bytes) 000618 1786989108.942518 fallocate 0x03354d thru 0x03db0e (0xa5c2 bytes) 000619 1786989108.950872 mapread 0x020114 thru 0x025e3a (0x5d27 bytes) 000620 1786989108.995382 write 0x02bd21 thru 0x0346f0 (0x89d0 bytes) 000621 1786989109.010008 mapwrite 0x00dbe0 thru 0x00e271 (0x692 bytes) 000622 1786989109.042138 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3db0f): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db0f): Mirror 16: 0x9e54f3b8 (stale) Mirror 17: 0x9e54f3b8 (stale) Mirror 18: 0x9e54f3b8 (stale) Mirror 19: 0x9e54f3b8 (stale) CRC-32 checksum value for chunk [0, 0x3db0f): Mirror 1: 0xc075b1da Mirror 20: 0xc075b1da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db0f: skipped 000623 1786989110.097413 trunc from 0x0360c8 to 0x03941d (0x3356 bytes) 000624 1786989110.153100 trunc from 0x007035 to 0x015f3a (0xef06 bytes) 000625 1786989110.162712 trunc from 0x011d8e to 0x01f781 (0xd9f4 bytes) 000626 1786989110.175418 mapread 0x016252 thru 0x0260ba (0xfe69 bytes) 000627 1786989110.218814 punch from 0x030159 to 0x03db0e (0xd9b6 bytes) 000628 1786989110.229972 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3db0f): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db0f): Mirror 16: 0x9e54f3b8 (stale) Mirror 17: 0x9e54f3b8 (stale) Mirror 18: 0x9e54f3b8 (stale) Mirror 19: 0x9e54f3b8 (stale) Mirror 20: 0xc075b1da (stale) CRC-32 checksum value for chunk [0, 0x3db0f): Mirror 1: 0x58898d61 Mirror 21: 0x58898d61 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db0f: skipped 000629 1786989111.349382 read 0x00e64b thru 0x01de52 (0xf808 bytes) 000630 1786989111.373805 write 0x012ac2 thru 0x013801 (0xd40 bytes) 000632 1786989111.471037 trunc from 0x03b05f to 0x03bf85 (0xf27 bytes) 000633 1786989111.492773 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db0f): Mirror 16: 0x9e54f3b8 (stale) Mirror 17: 0x9e54f3b8 (stale) Mirror 19: 0x9e54f3b8 (stale) Mirror 20: 0xc075b1da (stale) Mirror 21: 0x58898d61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db0f): Mirror 16: 0x9e54f3b8 (stale) Mirror 17: 0x9e54f3b8 (stale) Mirror 19: 0x9e54f3b8 (stale) Mirror 20: 0xc075b1da (stale) Mirror 21: 0x58898d61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db0f: skipped 000634 1786989112.137200 fallocate 0x038d5d thru 0x03ffff (0x72a3 bytes) 000635 1786989112.145565 fallocate 0x00316c thru 0x009416 (0x62ab bytes) 000636 1786989112.151851 mapwrite 0x03dddc thru 0x03ffff (0x2224 bytes) 000637 1786989112.190083 mapwrite 0x016d22 thru 0x01e054 (0x7333 bytes) 000638 1786989112.228321 trunc from 0x03f60c to 0x03ffff (0x9f4 bytes) 000639 1786989112.235036 mapread 0x00ca2f thru 0x0108d4 (0x3ea6 bytes) 000640 1786989112.249082 write 0x020dfa thru 0x02ec02 (0xde09 bytes) 000644 1786989112.284834 mapread 0x01a147 thru 0x026368 (0xc222 bytes) 000645 1786989112.302433 punch from 0x01839a to 0x024f73 (0xcbda bytes) 000646 1786989112.339710 read 0x004da2 thru 0x00bcd2 (0x6f31 bytes) 000647 1786989112.346270 trunc from 0x021401 to 0x02892e (0x752e bytes) 000648 1786989112.352945 fallocate 0x038b0f thru 0x03f0d2 (0x65c4 bytes) 000649 1786989112.360322 punch from 0x01c194 to 0x025746 (0x95b3 bytes) 000650 1786989112.365794 mapwrite 0x02799e thru 0x0279f0 (0x53 bytes) 000651 1786989112.401003 punch from 0x01c899 to 0x02b50c (0xec74 bytes) 000652 1786989112.408264 fallocate 0x02a005 thru 0x03366d (0x9669 bytes) 000653 1786989112.415844 write 0x01013e thru 0x0158ba (0x577d bytes) 000654 1786989112.417901 write 0x001d6c thru 0x010730 (0xe9c5 bytes) 000655 1786989112.427255 write 0x00c85c thru 0x01149c (0x4c41 bytes) 000656 1786989112.430013 fallocate 0x03bf07 thru 0x03ffff (0x40f9 bytes) 000657 1786989112.437411 write 0x00022f thru 0x00f622 (0xf3f4 bytes) 000658 1786989112.444737 trunc from 0x01e377 to 0x0245a4 (0x622e bytes) 000659 1786989112.450697 punch from 0x036665 to 0x0383a6 (0x1d42 bytes) 000660 1786989112.456156 trunc from 0x040000 to 0x03798f (0x8670 bytes) 000661 1786989112.474649 trunc from 0x037990 to 0x00fd58 (0x27c37 bytes) 000663 1786989112.501968 read 0x00143d thru 0x00fd58 (0xe91c bytes) 000664 1786989112.503340 read 0x00adbf thru 0x00fd58 (0x4f9a bytes) 000665 1786989112.504163 trunc from 0x00fd59 to 0x03b658 (0x2b900 bytes) 000666 1786989112.529388 mapwrite 0x01b0b8 thru 0x02a9f3 (0xf93c bytes) 000667 1786989112.582061 mapread 0x010361 thru 0x01cad0 (0xc770 bytes) 000668 1786989112.592818 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b659): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b659): Mirror 16: 0x3906c79e (stale) Mirror 17: 0x3906c79e (stale) Mirror 19: 0x3906c79e (stale) Mirror 20: 0xfd8ac4d7 (stale) Mirror 21: 0xec90cec3 (stale) CRC-32 checksum value for chunk [0, 0x3b659): Mirror 1: 0xd3c186d1 Mirror 22: 0xd3c186d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b659: skipped 000669 1786989113.709622 mapread 0x004003 thru 0x00a0e9 (0x60e7 bytes) 000670 1786989113.737974 trunc from 0x0197ed to 0x026075 (0xc889 bytes) 000671 1786989113.790110 punch from 0x013765 to 0x0140e6 (0x982 bytes) 000672 1786989113.795391 read 0x02a290 thru 0x02c704 (0x2475 bytes) 000673 1786989113.801402 mapwrite 0x01edfb thru 0x02cb5b (0xdd61 bytes) 000674 1786989113.840722 mapwrite 0x025c9e thru 0x032337 (0xc69a bytes) 000675 1786989113.882095 fallocate 0x016a68 thru 0x0186a2 (0x1c3b bytes) 000676 1786989113.886890 punch from 0x01ad71 to 0x02093e (0x5bce bytes) 000677 1786989113.891518 mapwrite 0x012b7a thru 0x0178b3 (0x4d3a bytes) 000678 1786989113.936646 mapread 0x021307 thru 0x0309b8 (0xf6b2 bytes) 000679 1786989113.937974 trunc from 0x00d80b to 0x00f1c8 (0x19be bytes) 000680 1786989113.951121 write 0x002712 thru 0x0083cd (0x5cbc bytes) 000682 1786989113.976272 mapwrite 0x0250dd thru 0x033d56 (0xec7a bytes) 000684 1786989114.029015 fallocate 0x03cfb8 thru 0x03f9df (0x2a28 bytes) 000685 1786989114.036847 trunc from 0x00da49 to 0x016753 (0x8d0b bytes) 000686 1786989114.048241 read 0x006f1f thru 0x0085a4 (0x1686 bytes) 000687 1786989114.049905 punch from 0x0262e2 to 0x02b1a3 (0x4ec2 bytes) 000688 1786989114.059024 write 0x03bd07 thru 0x03ffff (0x42f9 bytes) 000689 1786989114.087568 trunc from 0x0012ce to 0x007ce6 (0x6a19 bytes) 000690 1786989114.096993 fallocate 0x010ddb thru 0x013933 (0x2b59 bytes) 000691 1786989114.105223 mapwrite 0x0064b3 thru 0x0146fe (0xe24c bytes) 000692 1786989114.161043 punch from 0x017388 to 0x026b53 (0xf7cc bytes) 000693 1786989114.180662 punch from 0x025660 to 0x028eb2 (0x3853 bytes) 000694 1786989114.191037 trunc from 0x040000 to 0x003c89 (0x3c376 bytes) 000696 1786989114.239958 mapwrite 0x0164bb thru 0x0201a3 (0x9ce9 bytes) 000697 1786989114.390943 write 0x02c082 thru 0x036e6e (0xaded bytes) 000698 1786989114.398493 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36e6f): Mirror 16: 0x866faab3 (stale) Mirror 17: 0x866faab3 (stale) Mirror 19: 0x866faab3 (stale) Mirror 21: 0x195785ec (stale) Mirror 22: 0x2ba5d4a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36e6f): Mirror 16: 0x866faab3 (stale) Mirror 17: 0x866faab3 (stale) Mirror 19: 0x866faab3 (stale) Mirror 21: 0x195785ec (stale) Mirror 22: 0x2ba5d4a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36e6f: skipped 000699 1786989114.972029 mapread 0x02de66 thru 0x031ad3 (0x3c6e bytes) 000700 1786989115.018854 fallocate 0x00cca5 thru 0x01077b (0x3ad7 bytes) 000701 1786989115.026891 punch from 0x013751 to 0x0140c8 (0x978 bytes) 000702 1786989115.035459 read 0x01cb0b thru 0x01f79b (0x2c91 bytes) 000703 1786989115.053647 write 0x022cc1 thru 0x022ec8 (0x208 bytes) 000704 1786989115.073337 read 0x00b3c9 thru 0x00dd33 (0x296b bytes) 000705 1786989115.101053 write 0x0044f1 thru 0x00cbc3 (0x86d3 bytes) 000706 1786989115.108108 trunc from 0x036e6f to 0x00a8f9 (0x2c575 bytes) 000707 1786989115.159784 fallocate 0x01da33 thru 0x026108 (0x86d6 bytes) 000708 1786989115.183432 write 0x01f2ce thru 0x024efd (0x5c30 bytes) 000709 1786989115.189940 punch from 0x01be3d to 0x024efd (0x90c1 bytes) 000710 1786989115.224959 write 0x03714a thru 0x037a38 (0x8ef bytes) 000711 1786989115.226210 read 0x018d69 thru 0x01b205 (0x249d bytes) 000712 1786989115.265756 trunc from 0x013636 to 0x021382 (0xdd4d bytes) 000713 1786989115.274810 fallocate 0x02808b thru 0x0312f9 (0x926f bytes) 000715 1786989115.289810 write 0x002078 thru 0x00fe85 (0xde0e bytes) 000716 1786989115.294053 trunc from 0x02a86d to 0x037a38 (0xd1cc bytes) 000717 1786989115.337527 trunc from 0x037a39 to 0x006851 (0x311e7 bytes) 000718 1786989115.377495 trunc from 0x006852 to 0x001eef (0x4962 bytes) 000719 1786989115.413296 punch from 0x00006d to 0x001eef (0x1e83 bytes) 000720 1786989115.422731 mapwrite 0x033790 thru 0x03ffff (0xc870 bytes) 000721 1786989115.642721 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xd2e52b26 (stale) Mirror 17: 0xd2e52b26 (stale) Mirror 19: 0xd2e52b26 (stale) Mirror 21: 0xa80fe66b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xd2e52b26 (stale) Mirror 17: 0xd2e52b26 (stale) Mirror 19: 0xd2e52b26 (stale) Mirror 21: 0xa80fe66b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000722 1786989116.152431 MIRROR_RESYNC 5 mirrors 000723 1786989117.289833 fallocate 0x0049b3 thru 0x00999a (0x4fe8 bytes) 000724 1786989117.350269 trunc from 0x040000 to 0x02265a (0x1d9a5 bytes) 000725 1786989117.370851 mapread 0x010477 thru 0x01e5c9 (0xe153 bytes) 000726 1786989117.410811 trunc from 0x02265b to 0x005a62 (0x1cbf8 bytes) 000727 1786989117.430524 write 0x01aa86 thru 0x01b2a9 (0x824 bytes) 000728 1786989117.431703 fallocate 0x0350c5 thru 0x03a111 (0x504d bytes) 000729 1786989117.441073 fallocate 0x03d531 thru 0x03ffff (0x2acf bytes) 000730 1786989117.457279 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x292ae4fe (stale) Mirror 17: 0x292ae4fe (stale) Mirror 19: 0x292ae4fe (stale) Mirror 21: 0x292ae4fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6cc8588a Mirror 22: 0x6cc8588a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000732 1786989118.595440 MIRROR_RESYNC 6 mirrors 000733 1786989120.179522 write 0x0176af thru 0x01f865 (0x81b7 bytes) 000736 1786989120.316382 read 0x01f63c thru 0x025108 (0x5acd bytes) 000737 1786989120.352112 punch from 0x03b470 to 0x03ffff (0x4b90 bytes) 000738 1786989120.362457 trunc from 0x033342 to 0x03ffff (0xccbe bytes) 000739 1786989120.371356 mapwrite 0x02bad9 thru 0x02c324 (0x84c bytes) 000740 1786989120.461873 mapwrite 0x00e44d thru 0x00f6ce (0x1282 bytes) 000741 1786989120.525179 read 0x013c74 thru 0x019a35 (0x5dc2 bytes) 000742 1786989120.525851 trunc from 0x03508c to 0x03b1e5 (0x615a bytes) 000744 1786989120.536826 trunc from 0x040000 to 0x01a046 (0x25fb9 bytes) 000745 1786989120.562810 trunc from 0x003048 to 0x00978c (0x6745 bytes) 000746 1786989120.578277 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a047): Mirror 17: 0x963881f4 (stale) Mirror 19: 0x963881f4 (stale) Mirror 21: 0x963881f4 (stale) Mirror 22: 0x963881f4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a047): Mirror 17: 0x963881f4 (stale) Mirror 19: 0x963881f4 (stale) Mirror 21: 0x963881f4 (stale) Mirror 22: 0x963881f4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a047: skipped 000747 1786989121.208598 mapread 0x00ab17 thru 0x011579 (0x6a63 bytes) 000748 1786989121.302070 punch from 0x0004fe to 0x00f44a (0xef4d bytes) 000749 1786989121.311034 punch from 0x00215e to 0x00952f (0x73d2 bytes) 000750 1786989121.334557 mapwrite 0x02969c thru 0x0318cb (0x8230 bytes) 000751 1786989121.445578 fallocate 0x028693 thru 0x03439b (0xbd09 bytes) 000752 1786989121.453034 fallocate 0x02a9af thru 0x02c113 (0x1765 bytes) 000753 1786989121.466548 read 0x01dcb4 thru 0x02dad6 (0xfe23 bytes) 000754 1786989121.500805 mapread 0x02810e thru 0x02ef54 (0x6e47 bytes) 000755 1786989121.502055 write 0x00d2f5 thru 0x010f6d (0x3c79 bytes) 000757 1786989121.518753 punch from 0x00ea10 to 0x01be17 (0xd408 bytes) 000758 1786989121.538094 trunc from 0x03439c to 0x03d3b9 (0x901e bytes) 000759 1786989121.560836 trunc from 0x03d3ba to 0x0156b3 (0x27d06 bytes) 000760 1786989121.583587 read 0x00908e thru 0x00b30c (0x227f bytes) 000761 1786989121.611387 punch from 0x00c647 to 0x012ce6 (0x66a0 bytes) 000762 1786989121.615726 fallocate 0x00c0fa thru 0x00c8cb (0x7d2 bytes) 000763 1786989121.637378 trunc from 0x00616f to 0x00e074 (0x7f06 bytes) 000764 1786989121.670197 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x156b4): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x156b4): Mirror 17: 0xbdf9f199 (stale) Mirror 19: 0xbdf9f199 (stale) Mirror 21: 0xbdf9f199 (stale) Mirror 22: 0xbdf9f199 (stale) CRC-32 checksum value for chunk [0, 0x156b4): Mirror 1: 0xbdf9f199 Mirror 23: 0xbdf9f199 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x156b4: skipped 000765 1786989123.251569 write 0x0190ea thru 0x028b9b (0xfab2 bytes) 000766 1786989123.355016 write 0x007a59 thru 0x015d47 (0xe2ef bytes) 000767 1786989123.385540 trunc from 0x023447 to 0x028b9b (0x5755 bytes) 000769 1786989123.447168 write 0x03eea6 thru 0x03ffff (0x115a bytes) 000770 1786989123.461643 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x6cc8588a (stale) Mirror 19: 0x6cc8588a (stale) Mirror 21: 0x6cc8588a (stale) Mirror 23: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x6cc8588a (stale) Mirror 19: 0x6cc8588a (stale) Mirror 21: 0x6cc8588a (stale) Mirror 23: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000771 1786989124.113694 mapwrite 0x02d755 thru 0x0356b9 (0x7f65 bytes) 000773 1786989124.161764 mapread 0x0068bd thru 0x01549b (0xebdf bytes) 000775 1786989124.185763 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x6cc8588a (stale) Mirror 21: 0x6cc8588a (stale) Mirror 23: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0x6cc8588a (stale) Mirror 21: 0x6cc8588a (stale) Mirror 23: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000776 1786989124.587311 read 0x0059a6 thru 0x014a0a (0xf065 bytes) 000777 1786989124.655838 mapwrite 0x02c2f2 thru 0x032d86 (0x6a95 bytes) 000778 1786989124.723839 fallocate 0x017a61 thru 0x01a6d9 (0x2c79 bytes) 000779 1786989124.736820 mapread 0x004e35 thru 0x01022d (0xb3f9 bytes) 000780 1786989124.749192 mapwrite 0x00e6c5 thru 0x01da67 (0xf3a3 bytes) 000781 1786989124.781863 trunc from 0x02fa51 to 0x03a354 (0xa904 bytes) 000782 1786989124.789276 punch from 0x03ad23 to 0x03ffff (0x52dd bytes) 000783 1786989124.794137 mapwrite 0x029185 thru 0x02d577 (0x43f3 bytes) 000784 1786989124.818621 read 0x022612 thru 0x022737 (0x126 bytes) 000785 1786989124.840284 punch from 0x03d274 to 0x03ffff (0x2d8c bytes) 000786 1786989124.846026 write 0x02ae9b thru 0x031b3d (0x6ca3 bytes) 000787 1786989124.854544 punch from 0x0263dc to 0x02ab6d (0x4792 bytes) 000788 1786989124.870238 read 0x006c44 thru 0x00e4d7 (0x7894 bytes) 000789 1786989124.871270 fallocate 0x03371a thru 0x03ffff (0xc8e6 bytes) 000790 1786989124.877049 read 0x01f463 thru 0x02bcb6 (0xc854 bytes) 000791 1786989124.889233 trunc from 0x040000 to 0x03001b (0xffe4 bytes) 000792 1786989124.913626 punch from 0x010d7c to 0x0195fe (0x8883 bytes) 000793 1786989124.921006 trunc from 0x018a84 to 0x01af2e (0x24ab bytes) 000796 1786989124.934904 fallocate 0x019910 thru 0x0283a6 (0xea97 bytes) 000797 1786989124.942646 read 0x02aea2 thru 0x03001b (0x517a bytes) 000798 1786989124.944675 trunc from 0x013195 to 0x01897e (0x57ea bytes) 000799 1786989124.953235 write 0x035651 thru 0x03e358 (0x8d08 bytes) 000800 1786989124.956467 trunc from 0x03e359 to 0x01f325 (0x1f033 bytes) 000801 1786989124.980874 write 0x00ec5c thru 0x01c53a (0xd8df bytes) 000802 1786989124.989795 write 0x03281c thru 0x036b03 (0x42e8 bytes) 000804 1786989124.996235 trunc from 0x036b04 to 0x0353cf (0x1734 bytes) 000805 1786989125.026875 trunc from 0x026ab7 to 0x031a09 (0xaf53 bytes) 000806 1786989125.035263 mapwrite 0x022926 thru 0x02ac59 (0x8334 bytes) 000807 1786989125.079767 punch from 0x01a137 to 0x027f68 (0xde32 bytes) 000808 1786989125.089102 fallocate 0x032c95 thru 0x03ffff (0xd36b bytes) 000809 1786989125.095069 fallocate 0x025d5d thru 0x02da46 (0x7cea bytes) 000810 1786989125.103675 trunc from 0x040000 to 0x0321b5 (0xde4a bytes) 000811 1786989125.124531 trunc from 0x00bdac to 0x012c60 (0x6eb5 bytes) 000812 1786989125.146003 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x321b6): Mirror 19: 0x8f0e31a8 (stale) Mirror 23: 0x16e3b4e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x321b6): Mirror 19: 0x8f0e31a8 (stale) Mirror 23: 0x16e3b4e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x321b6: skipped 000814 1786989125.810763 trunc from 0x010a9e to 0x017142 (0x66a5 bytes) 000817 1786989125.826001 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x321b6): Mirror 23: 0x16e3b4e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x321b6): Mirror 23: 0x16e3b4e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x321b6: skipped 000818 1786989126.127767 read 0x00986f thru 0x00bd44 (0x24d6 bytes) 000819 1786989126.144241 write 0x001477 thru 0x010953 (0xf4dd bytes) 000820 1786989126.170327 read 0x030290 thru 0x031e42 (0x1bb3 bytes) 000821 1786989126.188580 read 0x021d34 thru 0x029178 (0x7445 bytes) 000822 1786989126.197101 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000823 1786989126.406614 punch from 0x01bcff to 0x027555 (0xb857 bytes) 000824 1786989126.424634 punch from 0x00c9fe to 0x01b049 (0xe64c bytes) 000825 1786989126.431800 trunc from 0x0321b6 to 0x003393 (0x2ee22 bytes) 000826 1786989126.462474 fallocate 0x032e0a thru 0x03aa3b (0x7c32 bytes) 000827 1786989126.472616 mapread 0x00192e thru 0x00220e (0x8e1 bytes) 000828 1786989126.485586 read 0x0355b2 thru 0x03aa3b (0x548a bytes) 000829 1786989126.496817 mapread 0x032a19 thru 0x0389e4 (0x5fcc bytes) 000830 1786989126.517663 MIRROR_EXTEND to 2 mirrors 000831 1786989127.521212 write 0x0041e6 thru 0x011214 (0xd02f bytes) 000832 1786989127.598750 mapwrite 0x00df06 thru 0x01925a (0xb355 bytes) 000833 1786989127.643769 mapread 0x002c21 thru 0x003db3 (0x1193 bytes) 000834 1786989127.656897 fallocate 0x010d03 thru 0x01b0d7 (0xa3d5 bytes) 000835 1786989127.666249 mapread 0x019575 thru 0x026ce9 (0xd775 bytes) 000836 1786989127.688305 read 0x02431b thru 0x02dea8 (0x9b8e bytes) 000837 1786989127.689180 punch from 0x01412b to 0x01b620 (0x74f6 bytes) 000838 1786989127.695536 punch from 0x0076dd to 0x00d449 (0x5d6d bytes) 000839 1786989127.702114 fallocate 0x032f03 thru 0x03a661 (0x775f bytes) 000840 1786989127.712482 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000841 1786989127.852123 read 0x02c274 thru 0x033c79 (0x7a06 bytes) 000842 1786989127.869229 trunc from 0x01e15d to 0x01ff61 (0x1e05 bytes) 000843 1786989127.879253 fallocate 0x021617 thru 0x030cfc (0xf6e6 bytes) 000844 1786989127.885901 mapread 0x033d9b thru 0x03aa3b (0x6ca1 bytes) 000846 1786989127.908968 trunc from 0x02bd99 to 0x02c5cf (0x837 bytes) 000849 1786989127.916890 mapread 0x0054d2 thru 0x0066ca (0x11f9 bytes) 000850 1786989127.939248 mapwrite 0x02bb0a thru 0x02e0f3 (0x25ea bytes) 000851 1786989127.972036 read 0x01cfa2 thru 0x021812 (0x4871 bytes) 000852 1786989127.984583 mapread 0x02e96b thru 0x03965b (0xacf1 bytes) 000853 1786989127.986657 mapwrite 0x0399ab thru 0x03ffff (0x6655 bytes) 000854 1786989128.038606 write 0x00a092 thru 0x014369 (0xa2d8 bytes) 000855 1786989128.040696 mapread 0x0265aa thru 0x02aaf2 (0x4549 bytes) 000856 1786989128.042297 punch from 0x007a69 to 0x00f76f (0x7d07 bytes) 000857 1786989128.059511 write 0x0380ea thru 0x03d91d (0x5834 bytes) 000858 1786989128.060957 trunc from 0x028001 to 0x0314ae (0x94ae bytes) 000859 1786989128.066822 trunc from 0x040000 to 0x0292e7 (0x16d18 bytes) 000861 1786989128.102210 trunc from 0x0114d8 to 0x0124b9 (0xfe2 bytes) 000862 1786989128.108222 punch from 0x018b78 to 0x022354 (0x97dd bytes) 000863 1786989128.115694 fallocate 0x00ec3f thru 0x00fd38 (0x10fa bytes) 000864 1786989128.123079 mapwrite 0x021c58 thru 0x027320 (0x56c9 bytes) 000865 1786989128.205292 trunc from 0x01aece to 0x01af47 (0x7a bytes) 000866 1786989128.210000 write 0x00cfb1 thru 0x01c6a1 (0xf6f1 bytes) 000869 1786989128.229261 mapwrite 0x03e27b thru 0x03ffff (0x1d85 bytes) 000870 1786989128.288180 trunc from 0x040000 to 0x015b19 (0x2a4e6 bytes) 000871 1786989128.309430 mapwrite 0x01b174 thru 0x01beb6 (0xd43 bytes) 000872 1786989128.357818 fallocate 0x000e7f thru 0x008063 (0x71e5 bytes) 000873 1786989128.364127 punch from 0x00dada to 0x00e775 (0xc9c bytes) 000874 1786989128.370180 write 0x036f18 thru 0x03e0c1 (0x71aa bytes) 000875 1786989128.373420 mapwrite 0x0093cb thru 0x012b14 (0x974a bytes) 000876 1786989128.457460 write 0x02af4c thru 0x037de8 (0xce9d bytes) 000877 1786989128.474488 read 0x007582 thru 0x00fe89 (0x8908 bytes) 000878 1786989128.503879 MIRROR_EXTEND to 2 mirrors 000879 1786989129.271370 write 0x01350b thru 0x0184eb (0x4fe1 bytes) 000880 1786989129.359515 fallocate 0x03fff4 thru 0x03ffff (0xc bytes) 000882 1786989129.378449 trunc from 0x025fbc to 0x02c31d (0x6362 bytes) 000883 1786989129.390000 mapwrite 0x0090ec thru 0x00ba87 (0x299c bytes) 000884 1786989129.443083 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb2332ec (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb18d850 Mirror 3: 0xb18d850 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000885 1786989130.470887 mapwrite 0x002b6c thru 0x011c33 (0xf0c8 bytes) 000886 1786989130.631197 trunc from 0x040000 to 0x03e4f9 (0x1b06 bytes) 000887 1786989130.689544 mapread 0x031bf9 thru 0x03cf59 (0xb361 bytes) 000888 1786989130.707820 punch from 0x00dc92 to 0x01bd42 (0xe0b1 bytes) 000889 1786989130.713240 trunc from 0x010c89 to 0x01b773 (0xaaeb bytes) 000890 1786989130.717571 write 0x01deb0 thru 0x025f89 (0x80da bytes) 000891 1786989130.725839 trunc from 0x03e4fa to 0x002e79 (0x3b680 bytes) 000892 1786989130.743617 mapread 0x00090a thru 0x002e79 (0x2570 bytes) 000893 1786989130.750137 mapread 0x0014fb thru 0x002e79 (0x197f bytes) 000894 1786989130.751843 punch from 0x001481 to 0x002e79 (0x19f9 bytes) 000895 1786989130.758246 read 0x000374 thru 0x002e79 (0x2b06 bytes) 000896 1786989130.763654 trunc from 0x002e7a to 0x008f43 (0x60ca bytes) 000897 1786989130.782331 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8f44): Mirror 3: 0xa148f93a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8f44): Mirror 3: 0xa148f93a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8f44: skipped 000898 1786989131.103628 mapread 0x008832 thru 0x008f43 (0x712 bytes) 000899 1786989131.114292 read 0x002db6 thru 0x007eb4 (0x50ff bytes) 000900 1786989131.158822 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x8f44): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8f44): Mirror 3: 0xa148f93a (stale) CRC-32 checksum value for chunk [0, 0x8f44): Mirror 1: 0x75a8b5ba Mirror 4: 0x75a8b5ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8f44: skipped 000902 1786989132.650169 fallocate 0x00726d thru 0x00d4a4 (0x6238 bytes) 000903 1786989132.706600 punch from 0x004de5 to 0x00d4a4 (0x86c0 bytes) 000904 1786989132.714469 write 0x01d1c3 thru 0x021356 (0x4194 bytes) 000905 1786989132.728236 trunc from 0x021357 to 0x0367e1 (0x1548b bytes) 000906 1786989132.751871 fallocate 0x038a43 thru 0x03ffff (0x75bd bytes) 000907 1786989132.757744 trunc from 0x040000 to 0x03ebc3 (0x143c bytes) 000908 1786989132.785869 mapwrite 0x036242 thru 0x03ffff (0x9dbe bytes) 000910 1786989132.921422 mapwrite 0x029465 thru 0x034d05 (0xb8a1 bytes) 000911 1786989133.025057 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x85758b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x85758b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000912 1786989133.470630 mapwrite 0x009c54 thru 0x0142da (0xa687 bytes) 000913 1786989133.573780 punch from 0x028e96 to 0x02db00 (0x4c6b bytes) 000914 1786989133.587709 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000915 1786989133.852472 punch from 0x00bfe3 to 0x00c7e7 (0x805 bytes) 000916 1786989133.860723 trunc from 0x040000 to 0x03f82d (0x7d2 bytes) 000917 1786989133.906813 mapread 0x00a0b3 thru 0x00cfe3 (0x2f31 bytes) 000918 1786989133.968278 fallocate 0x0128aa thru 0x01ab1b (0x8272 bytes) 000920 1786989133.991497 mapread 0x02eabc thru 0x033599 (0x4ade bytes) 000921 1786989134.030436 trunc from 0x015b09 to 0x025092 (0xf58a bytes) 000922 1786989134.039887 trunc from 0x01834e to 0x024c60 (0xc913 bytes) 000924 1786989134.055597 fallocate 0x0115c3 thru 0x0185cd (0x700b bytes) 000926 1786989134.070401 read 0x0382ec thru 0x03f761 (0x7476 bytes) 000927 1786989134.086950 trunc from 0x007ea0 to 0x012b4f (0xacb0 bytes) 000928 1786989134.099096 mapwrite 0x01832f thru 0x025717 (0xd3e9 bytes) 000929 1786989134.160847 fallocate 0x01daa3 thru 0x020990 (0x2eee bytes) 000930 1786989134.173194 fallocate 0x03de35 thru 0x03ffff (0x21cb bytes) 000932 1786989134.200086 mapread 0x03baa7 thru 0x03ffff (0x4559 bytes) 000933 1786989134.202839 read 0x03b4a6 thru 0x03ffff (0x4b5a bytes) 000934 1786989134.204394 trunc from 0x010945 to 0x01ce14 (0xc4d0 bytes) 000935 1786989134.209847 read 0x03f42e thru 0x03ffff (0xbd2 bytes) 000936 1786989134.211588 fallocate 0x02faa9 thru 0x03b13a (0xb692 bytes) 000937 1786989134.218768 trunc from 0x040000 to 0x0017bb (0x3e844 bytes) 000938 1786989134.244015 trunc from 0x001257 to 0x0017bb (0x565 bytes) 000939 1786989134.262157 punch from 0x000e1d to 0x0017bb (0x99f bytes) 000941 1786989134.282264 trunc from 0x000fc7 to 0x0017bb (0x7f5 bytes) 000942 1786989134.301438 mapwrite 0x023f53 thru 0x0329cc (0xea7a bytes) 000943 1786989134.348613 trunc from 0x0329cd to 0x0369a3 (0x3fd7 bytes) 000944 1786989134.357386 read 0x02da8c thru 0x0369a3 (0x8f18 bytes) 000945 1786989134.362632 write 0x006528 thru 0x00f659 (0x9132 bytes) 000946 1786989134.365058 trunc from 0x015b74 to 0x021315 (0xb7a2 bytes) 000947 1786989134.371451 mapread 0x0212ff thru 0x026e77 (0x5b79 bytes) 000948 1786989134.375833 write 0x0197f7 thru 0x01b510 (0x1d1a bytes) 000949 1786989134.381685 write 0x0200fe thru 0x02c03f (0xbf42 bytes) 000950 1786989134.386545 fallocate 0x0317a4 thru 0x033ccb (0x2528 bytes) 000951 1786989134.390449 fallocate 0x024cfb thru 0x02d37b (0x8681 bytes) 000953 1786989134.394595 write 0x037f58 thru 0x03ffff (0x80a8 bytes) 000955 1786989134.396508 punch from 0x028cec to 0x028d9b (0xb0 bytes) 000956 1786989134.407703 mapread 0x01f1b0 thru 0x02ac3d (0xba8e bytes) 000957 1786989134.414481 fallocate 0x014fd2 thru 0x021a0f (0xca3e bytes) 000958 1786989134.418275 trunc from 0x040000 to 0x02ec38 (0x113c7 bytes) 000959 1786989134.431238 trunc from 0x02ec39 to 0x037de5 (0x91ad bytes) 000960 1786989134.439494 read 0x00895b thru 0x009435 (0xadb bytes) 000962 1786989134.440532 fallocate 0x001bc3 thru 0x00aae8 (0x8f26 bytes) 000965 1786989134.445277 punch from 0x00772c to 0x017304 (0xfbd9 bytes) 000966 1786989134.449340 mapwrite 0x013384 thru 0x014557 (0x11d4 bytes) 000967 1786989134.462635 read 0x036146 thru 0x037de5 (0x1ca0 bytes) 000968 1786989134.463330 trunc from 0x037de6 to 0x008168 (0x2fc7d bytes) 000969 1786989134.479869 trunc from 0x008169 to 0x02cb87 (0x24a1f bytes) 000971 1786989134.494798 trunc from 0x02721f to 0x02aa0d (0x37ef bytes) 000972 1786989134.499335 write 0x0078fd thru 0x011e64 (0xa568 bytes) 000973 1786989134.509904 read 0x0293c8 thru 0x02cb87 (0x37c0 bytes) 000975 1786989134.518709 fallocate 0x0021f1 thru 0x00e008 (0xbe18 bytes) 000976 1786989134.524341 MIRROR_EXTEND to 2 mirrors 000977 1786989135.484010 trunc from 0x007ae8 to 0x00acab (0x31c4 bytes) 000978 1786989135.566246 write 0x00581e thru 0x00bde0 (0x65c3 bytes) 000979 1786989135.590764 punch from 0x00e78d to 0x012610 (0x3e84 bytes) 000980 1786989135.600432 trunc from 0x016170 to 0x021836 (0xb6c7 bytes) 000981 1786989135.608111 mapread 0x02213b thru 0x02ad43 (0x8c09 bytes) 000982 1786989135.635382 punch from 0x01772c to 0x022a38 (0xb30d bytes) 000983 1786989135.642146 write 0x022deb thru 0x029d38 (0x6f4e bytes) 000984 1786989135.652414 trunc from 0x02cb88 to 0x02b47f (0x1708 bytes) 000986 1786989135.687780 punch from 0x00f9f5 to 0x01dc17 (0xe223 bytes) 000987 1786989135.697705 read 0x00a39c thru 0x00e2b1 (0x3f16 bytes) 000988 1786989135.707075 read 0x02544a thru 0x02b47f (0x6036 bytes) 000989 1786989135.708885 punch from 0x00307e to 0x00b482 (0x8405 bytes) 000990 1786989135.731979 write 0x001fb4 thru 0x0087f2 (0x683f bytes) 000991 1786989135.763163 write 0x03b96e thru 0x03ffff (0x4692 bytes) 000992 1786989135.773186 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000993 1786989135.966852 punch from 0x0342ab to 0x03efa7 (0xacfd bytes) 000994 1786989135.974516 write 0x033d0f thru 0x03b6e6 (0x79d8 bytes) 000995 1786989136.000173 MIRROR_EXTEND to 2 mirrors 000996 1786989137.098816 trunc from 0x0069e9 to 0x0161ee (0xf806 bytes) 000997 1786989137.127668 punch from 0x01ff50 to 0x028260 (0x8311 bytes) 000998 1786989137.141131 read 0x01989f thru 0x028398 (0xeafa bytes) 000999 1786989137.159864 punch from 0x00013e to 0x009b56 (0x9a19 bytes) 001000 1786989137.167714 mapwrite 0x03e043 thru 0x03f3ab (0x1369 bytes) All operations completed A-OK! PASS 70a (123s) == sanity-flr test 71: check mirror extend parent fid ==== 13:52:23 (1786989143) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0xa5:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 13:52:36 (1786989156) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.240596 s, 8.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136833 s, 7.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 13:52:46 (1786989166) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5524256 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6211712 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8508512 .. Extending file size to 8218848 .. Extending file size to 8785888 .. Extending file size to 8750048 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8806208 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62064 62065 62067 62069 62070 done 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (91s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:54:17 (1786989257) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8355680 .. Extending file size to 3760000 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8566112 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8651040 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9077824 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63362 63363 63364 63366 63368 done 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:55:38 (1786989338) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2184 1269120 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2196 1269108 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18728 3588292 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24568 3582452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43296 7170744 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2184 1269120 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2196 1269108 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18728 3588292 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24568 3582452 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43296 7170744 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150841 s, 13.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23648 s, 8.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xcb6183e2 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:56:00 (1786989360) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.299424 s, 7.0 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 13:56:09 (1786989369) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:56:19 (1786989379) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17736 s, 11.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:56:29 (1786989389) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108966 s, 19.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:56:40 (1786989400) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0932064 s, 22.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22d:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:57:15 (1786989435) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647104 s, 16.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x22e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x22d:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:58:11 (1786989491) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156921 s, 13.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:59:03 (1786989543) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg222-server: Pool lustre.test_204f created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.217405 s, 9.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg222-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:00:21 (1786989621) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:00:28 (1786989628) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:00:36 (1786989636) oleg222-server: Pool lustre.test_206 created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg222-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:01:07 (1786989667) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140314 s, 7.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552495 s, 19.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x65494f04 Mirror 3: 0x65494f04 PASS 207 (17s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:01:25 (1786989685) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (6s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:01:31 (1786989691) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:01:35 (1786989695) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0748546 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x23b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x8e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x8e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x8e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12d0000 Read this (4 bytes): BBBB 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 209a (47s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:02:22 (1786989742) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.016035 s, 511 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:02:31 (1786989751) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847271 s, 12.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd3 lmm_fid: [0x200000405:0xd3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd3 lmm_fid: [0x200000405:0xd3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611347 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-17 14:02:33.000000000 -0400 Modify: 2026-08-17 14:02:34.000000000 -0400 Change: 2026-08-17 14:02:35.000000000 -0400 Birth: 2026-08-17 14:02:34.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:02:42 (1786989762) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599297 s, 17.5 MB/s oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg222-server: O/280000401/d29/15d: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:03:06 (1786989786) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.347369 s, 118 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.023755 s, 172 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 17 14:03 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd8:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 17 14:03 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:03:16 (1786989796) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:03:24 (1786989804) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 2056 sec =========== 14:03:34 (1786989814) === sanity-flr: start cleanup 14:03:36 (1786989816) === === sanity-flr: finish cleanup 14:03:46 (1786989826) ===