-----============= acceptance-small: sanity-flr ============----- Thu Apr 30 14:09:59 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg342-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:10:05 (1777572605) === oleg342-client.virtnet: executing check_config_client /mnt/lustre oleg342-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg342-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88810ccbc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88810ccbc000.idle_timeout=debug disable quota as required oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:10:18 (1777572618) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7570] == sanity-flr test 0a: lfs mirror create with -N option == 14:10:18 (1777572618) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:10:23 (1777572623) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:10:25 (1777572625) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:10:26 (1777572626) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:10:34 (1777572634) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:10:35 (1777572635) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 14:10:37 (1777572637) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88810ccbc000.stats=clear osc.lustre-OST0001-osc-ffff88810ccbc000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:10:39 (1777572639) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:10:43 (1777572643) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:10:46 (1777572646) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:10:47 (1777572647) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:10:50 (1777572650) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 14:10:53 (1777572653) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:10:56 (1777572656) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:10:57 (1777572657) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:11:00 (1777572660) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:11:04 (1777572664) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:11:06 (1777572666) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.196265 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366927 s, 28.6 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:11:09 (1777572669) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.0828367 s, 114 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:11:12 (1777572672) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:11:16 (1777572676) Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88810ccbc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88810ccbc000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88810ccbc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88810ccbc000.ost_server_uuid in DISCONN state after 3 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg342-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:11:45 (1777572705) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0925029 s, 56.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88810ccbc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88810ccbc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:11:57 (1777572717) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88810ccbc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88810ccbc000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88810ccbc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88810ccbc000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg342-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (24s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:12:21 (1777572741) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.09003 s, 48.1 MB/s -rw-r--r-- 1 root root 52428800 Apr 30 14:12 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.255s user 0m0.000s sys 0m0.139s Stopping /mnt/lustre-ost1 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88810ccbc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88810ccbc000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.243s user 0m0.000s sys 0m0.130s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg342-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg342-server oleg342-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88810ccbc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88810ccbc000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.261s user 0m0.001s sys 0m0.135s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg342-server: oleg342-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg342-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg342-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88810ccbc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88810ccbc000.ost_server_uuid in FULL state after 0 sec PASS 33b (29s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:12:50 (1777572770) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:12:51 (1777572771) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:12:52 (1777572772) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 14:12:53 (1777572773) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 14:12:55 (1777572775) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.16291 s, 77.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:12:59 (1777572779) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130181 s, 80.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109934 s, 95.4 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:13:01 (1777572781) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:13:05 (1777572785) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000568746 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00069147 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000565831 s, 1.8 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00058954 s, 1.7 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 14:13:08 (1777572788) create test files with size 15729256 .. 4186763601e191b24d032b78e7701b9f - 769c0f5fbbbaffc01639c6806853f78c - ca08d4c9d1b763a64d019f3aacd003e6 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg342-client.virtnet /mnt/lustre (opts:) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 14:13:13 (1777572793) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:13:17 (1777572797) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:13:19 (1777572799) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446207 s, 23.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121906 s, 86.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 14:13:22 (1777572802) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0368611 s, 56.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0896256 s, 46.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0475675 s, 44.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 14:13:26 (1777572806) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 14:13:27 (1777572807) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:13:28 (1777572808) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00958611 s, 109 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 14:13:30 (1777572810) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:13:31 (1777572811) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:13:34 (1777572814) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0853583 s, 123 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg342-client.virtnet /mnt/lustre (opts:) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (21s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:13:55 (1777572835) llite.lustre-ffff8881181aa800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0845296 s, 124 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg342-client.virtnet /mnt/lustre (opts:) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8881092f5800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:13:59 (1777572839) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (11s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:14:10 (1777572850) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:14:20 (1777572860) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 14:14:21 (1777572861) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0736151 s, 42.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.05944 s, 52.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x318403da (stale) Mirror 2: 0x318403da (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 3856b3c08cb66a49ec20bc43d2a7d8e9 - mirror 1 checksum: 14161b59120e2ce46243be19531471e9 - mirror 2 checksum: 3856b3c08cb66a49ec20bc43d2a7d8e9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 30 14:14:22 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Apr 30 14:14:22 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3856b3c08cb66a49ec20bc43d2a7d8e9 - mirror 1 checksum: 14161b59120e2ce46243be19531471e9 - mirror 2 checksum: 14161b59120e2ce46243be19531471e9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 30 14:14:22 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Apr 30 14:14:22 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:14:24 (1777572864) Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 14:14:26 (1777572866) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 14:14:40 (1777572880) targets are mounted 14:14:40 (1777572880) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114078 s, 91.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (22s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:14:46 (1777572886) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:14:47 (1777572887) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:14:48 (1777572888) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00224702 s, 7.3 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:14:50 (1777572890) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:14:51 (1777572891) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:14:52 (1777572892) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:14:57 (1777572897) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:15:16 (1777572916) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:15:34 (1777572934) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (17s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:15:51 (1777572951) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 14:15:54 (1777572954) mirror create pid 53489 mirror split pid 53490 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53489 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53490 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 14:16:56 (1777573016) Seed set to 3016 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000007 1777573016.894092 write 0x018a20 thru 0x01f70e (0x6cef bytes) 000008 1777573016.895140 mapwrite 0x03830a thru 0x03ffff (0x7cf6 bytes) truncating to largest ever: 0x70c 000010 1777573016.923996 trunc from 0x040000 to 0x00070b (0x3f8f4 bytes) 000011 1777573016.928713 mapwrite 0x03cbb3 thru 0x03ffff (0x344d bytes) 000014 1777573016.949127 read 0x024a4e thru 0x02ba5d (0x7010 bytes) 000016 1777573016.952783 write 0x00e2e7 thru 0x0179fc (0x9716 bytes) 000021 1777573016.959027 mapwrite 0x03c02a thru 0x03ffff (0x3fd6 bytes) truncating to largest ever: 0x37745 000024 1777573016.974109 trunc from 0x040000 to 0x037744 (0x88bb bytes) truncating to largest ever: 0x3e523 000025 1777573016.979453 trunc from 0x037745 to 0x03e522 (0x6dde bytes) 000026 1777573016.983342 trunc from 0x03e523 to 0x03cd3e (0x17e4 bytes) 000027 1777573016.987216 mapwrite 0x0121eb thru 0x015513 (0x3329 bytes) 000030 1777573016.999867 read 0x035690 thru 0x036d5e (0x16cf bytes) 000031 1777573017.000272 read 0x000c74 thru 0x001e9e (0x122b bytes) 000032 1777573017.004255 write 0x0318c2 thru 0x03ffff (0xe73e bytes) 000034 1777573017.005661 read 0x0245bc thru 0x028cfe (0x4743 bytes) 000035 1777573017.006073 read 0x00de78 thru 0x016795 (0x891e bytes) 000037 1777573017.006959 mapwrite 0x03f60f thru 0x03ffff (0x9f1 bytes) 000039 1777573017.018945 mapwrite 0x0269e3 thru 0x0291b7 (0x27d5 bytes) 000041 1777573017.032220 mapread 0x03cbe7 thru 0x03dc23 (0x103d bytes) 000043 1777573017.033679 mapread 0x0039e3 thru 0x0071eb (0x3809 bytes) 000044 1777573017.034606 read 0x02b040 thru 0x037748 (0xc709 bytes) 000045 1777573017.035232 trunc from 0x040000 to 0x02e2f5 (0x11d0a bytes) 000048 1777573017.041970 write 0x034ec1 thru 0x03a5d4 (0x5714 bytes) 000049 1777573017.043195 trunc from 0x03a5d5 to 0x008adf (0x31af5 bytes) 000050 1777573017.048978 mapwrite 0x03dd23 thru 0x03ffff (0x22dd bytes) 000051 1777573017.070874 read 0x01ea94 thru 0x02e5ec (0xfb59 bytes) 000052 1777573017.071640 read 0x008ffa thru 0x00cbe2 (0x3be9 bytes) 000053 1777573017.072231 mapread 0x03e18b thru 0x03ffff (0x1e75 bytes) 000056 1777573017.074157 trunc from 0x040000 to 0x000551 (0x3faae bytes) 000057 1777573017.080456 mapwrite 0x00a242 thru 0x019a10 (0xf7cf bytes) 000059 1777573017.105253 mapread 0x00c869 thru 0x019a10 (0xd1a8 bytes) 000060 1777573017.106447 mapread 0x015b2d thru 0x019a10 (0x3ee4 bytes) 000062 1777573017.107836 write 0x01a914 thru 0x021233 (0x6920 bytes) 000063 1777573017.109022 mapwrite 0x010a86 thru 0x013f5b (0x34d6 bytes) 000065 1777573017.125925 mapwrite 0x02b08b thru 0x03138f (0x6305 bytes) 000066 1777573017.151299 mapwrite 0x012d48 thru 0x02082f (0xdae8 bytes) 000067 1777573017.172851 mapwrite 0x02f9d7 thru 0x039812 (0x9e3c bytes) 000068 1777573017.195307 write 0x0296ab thru 0x034549 (0xae9f bytes) 000071 1777573017.197248 mapread 0x00a027 thru 0x00db73 (0x3b4d bytes) 000075 1777573017.199328 read 0x006c8a thru 0x01400c (0xd383 bytes) 000077 1777573017.200245 write 0x01a4c8 thru 0x01f23d (0x4d76 bytes) 000078 1777573017.201211 trunc from 0x039813 to 0x0122b4 (0x2755e bytes) 000080 1777573017.207384 trunc from 0x0122b5 to 0x002c70 (0xf644 bytes) 000081 1777573017.212342 write 0x012801 thru 0x01d38c (0xab8c bytes) 000082 1777573017.213829 write 0x0276bc thru 0x034f8c (0xd8d1 bytes) 000083 1777573017.215375 write 0x018fd0 thru 0x01e553 (0x5584 bytes) 000084 1777573017.219497 mapwrite 0x0185a4 thru 0x024f8d (0xc9ea bytes) 000086 1777573017.239091 read 0x01eceb thru 0x029e3b (0xb151 bytes) 000087 1777573017.239674 mapread 0x025954 thru 0x02b37f (0x5a2c bytes) 000088 1777573017.240442 read 0x01e4f3 thru 0x02db02 (0xf610 bytes) 000089 1777573017.240875 mapwrite 0x01a510 thru 0x01b859 (0x134a bytes) 000090 1777573017.257013 trunc from 0x034f8d to 0x01c4bd (0x18acf bytes) 000091 1777573017.263707 mapwrite 0x034168 thru 0x03bdb9 (0x7c52 bytes) 000093 1777573017.288026 mapwrite 0x03ced0 thru 0x03dd7d (0xeae bytes) 000094 1777573017.305554 mapread 0x01f0ce thru 0x02795c (0x888f bytes) 000095 1777573017.306320 mapread 0x0295f5 thru 0x037472 (0xde7e bytes) 000096 1777573017.307054 mapread 0x0177b4 thru 0x025a24 (0xe271 bytes) 000097 1777573017.307776 mapread 0x00f3e9 thru 0x019de9 (0xaa01 bytes) 000099 1777573017.308816 trunc from 0x03dd7e to 0x01ebd9 (0x1f1a4 bytes) 000105 1777573017.315486 read 0x012a21 thru 0x01ebd9 (0xc1b9 bytes) 000108 1777573017.316923 mapwrite 0x02c9b5 thru 0x0364a2 (0x9aee bytes) 000110 1777573017.338459 trunc from 0x0364a3 to 0x023b6a (0x12938 bytes) 000111 1777573017.343080 mapwrite 0x0146c9 thru 0x01a5f9 (0x5f31 bytes) 000116 1777573017.360093 write 0x03bb35 thru 0x03ffff (0x44cb bytes) 000125 1777573017.365743 read 0x03c189 thru 0x03ffff (0x3e77 bytes) 000126 1777573017.366433 read 0x000113 thru 0x0096bc (0x95aa bytes) 000127 1777573017.367048 mapread 0x03ed15 thru 0x03ffff (0x12eb bytes) 000128 1777573017.367868 write 0x034e18 thru 0x039cc5 (0x4eae bytes) 000132 1777573017.377827 mapread 0x037803 thru 0x03ffff (0x87fd bytes) 000134 1777573017.379186 mapread 0x02421d thru 0x02a6f4 (0x64d8 bytes) 000135 1777573017.379993 trunc from 0x040000 to 0x031cd9 (0xe326 bytes) 000137 1777573017.386842 mapread 0x008d88 thru 0x016f45 (0xe1be bytes) 000138 1777573017.387839 read 0x002af8 thru 0x00e9b4 (0xbebd bytes) 000139 1777573017.388301 read 0x00306c thru 0x0059ef (0x2984 bytes) 000143 1777573017.389866 mapread 0x00ef93 thru 0x011407 (0x2475 bytes) 000144 1777573017.390508 write 0x00460f thru 0x00e08a (0x9a7c bytes) 000146 1777573017.391921 mapread 0x03094a thru 0x031cd9 (0x1390 bytes) 000148 1777573017.393198 mapwrite 0x035cda thru 0x03c181 (0x64a8 bytes) 000149 1777573017.422379 mapread 0x022e1d thru 0x0267f9 (0x39dd bytes) 000151 1777573017.423606 read 0x023fd4 thru 0x02aac5 (0x6af2 bytes) 000153 1777573017.424699 read 0x010882 thru 0x01beb5 (0xb634 bytes) 000154 1777573017.425190 mapwrite 0x0060ee thru 0x0082c1 (0x21d4 bytes) 000158 1777573017.444843 trunc from 0x03c182 to 0x020d7a (0x1b407 bytes) 000159 1777573017.453004 MIRROR_EXTEND to 2 mirrors 000160 1777573017.611929 write 0x038636 thru 0x03ffff (0x79ca bytes) 000164 1777573017.630182 read 0x018703 thru 0x01d373 (0x4c71 bytes) 000166 1777573017.633025 mapread 0x01a685 thru 0x01d635 (0x2fb1 bytes) 000167 1777573017.635010 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdff4e448 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x43c00c3b Mirror 3: 0x43c00c3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000169 1777573017.872775 trunc from 0x040000 to 0x027e53 (0x181ac bytes) 000170 1777573017.896117 write 0x02674b thru 0x032d83 (0xc639 bytes) 000171 1777573017.898445 read 0x01bfd9 thru 0x021016 (0x503e bytes) 000172 1777573017.901180 mapread 0x02e510 thru 0x032d83 (0x4874 bytes) 000173 1777573017.902068 write 0x00f70e thru 0x0142f8 (0x4beb bytes) 000174 1777573017.905955 trunc from 0x032d84 to 0x01b23f (0x17b44 bytes) 000175 1777573017.911969 mapwrite 0x03b010 thru 0x03f27c (0x426d bytes) 000177 1777573017.931479 write 0x02fcf8 thru 0x032220 (0x2529 bytes) 000178 1777573017.934296 mapwrite 0x037748 thru 0x037d75 (0x62e bytes) 000179 1777573017.956220 mapwrite 0x02e28e thru 0x037ebc (0x9c2f bytes) 000180 1777573017.970694 mapwrite 0x014927 thru 0x017209 (0x28e3 bytes) 000181 1777573017.985979 mapread 0x02d38f thru 0x033a2d (0x669f bytes) 000182 1777573017.987048 read 0x000db0 thru 0x00b60f (0xa860 bytes) 000183 1777573017.991096 trunc from 0x03f27d to 0x02a4b5 (0x14dc7 bytes) 000185 1777573017.997190 mapread 0x024cac thru 0x02a4b5 (0x580a bytes) 000186 1777573017.998249 read 0x01b01a thru 0x01bc12 (0xbf9 bytes) 000189 1777573017.999572 write 0x030be2 thru 0x0347e4 (0x3c03 bytes) 000190 1777573018.000353 trunc from 0x0347e5 to 0x03bb46 (0x7362 bytes) 000195 1777573018.007335 mapwrite 0x01ade6 thru 0x01e016 (0x3231 bytes) 000196 1777573018.022473 mapwrite 0x015ceb thru 0x01db5b (0x7e71 bytes) 000197 1777573018.037509 read 0x01107c thru 0x013a23 (0x29a8 bytes) 000200 1777573018.042283 read 0x01337b thru 0x020ee8 (0xdb6e bytes) 000201 1777573018.042906 read 0x035a2a thru 0x03bb46 (0x611d bytes) 000202 1777573018.046022 trunc from 0x03bb47 to 0x002cf4 (0x38e52 bytes) 000203 1777573018.051325 trunc from 0x002cf5 to 0x01c3ce (0x196da bytes) 000205 1777573018.056255 mapread 0x00647f thru 0x011bb9 (0xb73b bytes) 000206 1777573018.071502 mapread 0x00d94f thru 0x014eb7 (0x7569 bytes) 000207 1777573018.076020 write 0x002b36 thru 0x0051d2 (0x269d bytes) 000208 1777573018.078382 mapwrite 0x03eac5 thru 0x03ffff (0x153b bytes) 000209 1777573018.099104 mapwrite 0x0264b6 thru 0x02b6d8 (0x5223 bytes) 000210 1777573018.121853 trunc from 0x040000 to 0x01b37b (0x24c84 bytes) 000211 1777573018.126869 mapread 0x00e645 thru 0x0171cb (0x8b87 bytes) 000212 1777573018.131724 read 0x0024a0 thru 0x00972f (0x7290 bytes) 000213 1777573018.132521 mapread 0x00b117 thru 0x013c0d (0x8af7 bytes) 000214 1777573018.135158 MIRROR_RESYNC 3 mirrors 000218 1777573018.336791 MIRROR_EXTEND to 4 mirrors 000220 1777573018.529429 trunc from 0x01b37c to 0x008f0e (0x1246d bytes) 000221 1777573018.557912 mapread 0x001d54 thru 0x008f0e (0x71bb bytes) 000223 1777573018.565411 trunc from 0x008f0f to 0x01962e (0x10720 bytes) 000224 1777573018.571597 mapwrite 0x039ed2 thru 0x03d9b3 (0x3ae2 bytes) 000226 1777573018.591356 mapread 0x026b33 thru 0x034388 (0xd856 bytes) 000227 1777573018.597290 read 0x01d03e thru 0x02b8de (0xe8a1 bytes) 000228 1777573018.599360 trunc from 0x03d9b4 to 0x032534 (0xb47f bytes) 000229 1777573018.604356 trunc from 0x032535 to 0x01ba61 (0x16ad3 bytes) 000230 1777573018.609236 mapread 0x006fed thru 0x008e28 (0x1e3c bytes) 000231 1777573018.609995 write 0x004983 thru 0x01171a (0xcd98 bytes) 000232 1777573018.612499 trunc from 0x01ba62 to 0x00785a (0x14207 bytes) 000233 1777573018.617769 read 0x000746 thru 0x00785a (0x7115 bytes) 000235 1777573018.620639 trunc from 0x00785b to 0x00558e (0x22cc bytes) 000236 1777573018.626060 trunc from 0x00558f to 0x0275c4 (0x22036 bytes) 000237 1777573018.631296 mapread 0x002302 thru 0x010c5d (0xe95c bytes) 000238 1777573018.636989 mapwrite 0x03f82f thru 0x03ffff (0x7d1 bytes) 000239 1777573018.656298 write 0x03a95d thru 0x03dd89 (0x342d bytes) 000242 1777573018.660154 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf5730a1d (stale) Mirror 4: 0xf5730a1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf5730a1d (stale) Mirror 4: 0xf5730a1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000247 1777573018.768553 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf5730a1d (stale) Mirror 4: 0xf5730a1d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x50197a72 Mirror 5: 0x50197a72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000248 1777573019.043075 mapread 0x0006f3 thru 0x0061e6 (0x5af4 bytes) 000249 1777573019.052005 mapread 0x03105b thru 0x03b79e (0xa744 bytes) 000250 1777573019.059521 read 0x01bccf thru 0x01f667 (0x3999 bytes) 000251 1777573019.062709 mapread 0x01275b thru 0x012970 (0x216 bytes) 000253 1777573019.068110 mapwrite 0x000270 thru 0x00c71b (0xc4ac bytes) 000254 1777573019.122609 read 0x014c34 thru 0x017bc2 (0x2f8f bytes) 000255 1777573019.127583 write 0x00c3f4 thru 0x01474e (0x835b bytes) 000257 1777573019.128742 trunc from 0x040000 to 0x03850c (0x7af3 bytes) 000260 1777573019.141526 write 0x03c251 thru 0x03ffff (0x3daf bytes) 000261 1777573019.142355 trunc from 0x040000 to 0x017536 (0x28ac9 bytes) 000262 1777573019.148481 read 0x0152bb thru 0x017536 (0x227c bytes) 000263 1777573019.149059 write 0x030f66 thru 0x031150 (0x1eb bytes) 000264 1777573019.151660 MIRROR_RESYNC 4 mirrors 000265 1777573019.408561 read 0x00edb4 thru 0x01cdae (0xdffb bytes) 000267 1777573019.413453 MIRROR_EXTEND to 5 mirrors 000270 1777573019.607624 write 0x0204e6 thru 0x0267ab (0x62c6 bytes) 000273 1777573019.632685 trunc from 0x031151 to 0x028451 (0x8cff bytes) 000274 1777573019.646328 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28452): Mirror 4: 0xa818b9e7 (stale) Mirror 5: 0xa818b9e7 (stale) Mirror 6: 0xa818b9e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28452): Mirror 4: 0xa818b9e7 (stale) Mirror 5: 0xa818b9e7 (stale) Mirror 6: 0xa818b9e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28452: skipped 000276 1777573019.757281 mapread 0x005974 thru 0x007434 (0x1ac1 bytes) 000277 1777573019.763971 mapwrite 0x025d04 thru 0x034784 (0xea81 bytes) 000278 1777573019.799000 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34785): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34785): Mirror 4: 0xc1d5f28c (stale) Mirror 5: 0xc1d5f28c (stale) Mirror 6: 0xc1d5f28c (stale) CRC-32 checksum value for chunk [0, 0x34785): Mirror 1: 0x5c6a80c3 Mirror 7: 0x5c6a80c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34785: skipped 000280 1777573020.060310 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34785): Mirror 4: 0xc1d5f28c (stale) Mirror 6: 0xc1d5f28c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34785): Mirror 4: 0xc1d5f28c (stale) Mirror 6: 0xc1d5f28c (stale) CRC-32 checksum value for chunk [0, 0x34785): Mirror 1: 0x5c6a80c3 Mirror 7: 0x5c6a80c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34785: skipped 000281 1777573020.249589 mapread 0x032a7e thru 0x034784 (0x1d07 bytes) 000282 1777573020.260455 MIRROR_RESYNC 4 mirrors 000283 1777573020.491688 MIRROR_RESYNC 4 mirrors 000284 1777573020.630558 mapread 0x032b5b thru 0x033168 (0x60e bytes) 000285 1777573020.634917 MIRROR_RESYNC 4 mirrors 000289 1777573020.766481 write 0x013c80 thru 0x021519 (0xd89a bytes) 000293 1777573020.788412 mapread 0x028415 thru 0x030921 (0x850d bytes) 000294 1777573020.794332 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34785): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34785): Mirror 4: 0x5c6a80c3 (stale) Mirror 6: 0x5c6a80c3 (stale) Mirror 7: 0x5c6a80c3 (stale) CRC-32 checksum value for chunk [0, 0x34785): Mirror 1: 0x557c3739 Mirror 8: 0x557c3739 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34785: skipped 000296 1777573021.032534 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34785): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34785): Mirror 4: 0x5c6a80c3 (stale) Mirror 6: 0x5c6a80c3 (stale) Mirror 7: 0x5c6a80c3 (stale) CRC-32 checksum value for chunk [0, 0x34785): Mirror 1: 0x557c3739 Mirror 8: 0x557c3739 Mirror 9: 0x557c3739 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34785: skipped 000297 1777573021.281039 trunc from 0x034785 to 0x02e1f2 (0x6592 bytes) 000298 1777573021.304994 trunc from 0x02e1f3 to 0x00dac0 (0x20732 bytes) 000299 1777573021.309732 mapwrite 0x02677a thru 0x0320b9 (0xb940 bytes) 000301 1777573021.334962 mapwrite 0x00c92c thru 0x00e878 (0x1f4d bytes) 000303 1777573021.349349 trunc from 0x0320ba to 0x02d1bb (0x4efe bytes) 000304 1777573021.355766 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d1bc): Mirror 6: 0xd3715a2a (stale) Mirror 7: 0xd3715a2a (stale) Mirror 8: 0x791109a8 (stale) Mirror 9: 0x791109a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d1bc): Mirror 6: 0xd3715a2a (stale) Mirror 7: 0xd3715a2a (stale) Mirror 8: 0x791109a8 (stale) Mirror 9: 0x791109a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d1bc: skipped 000306 1777573021.453481 mapwrite 0x00be32 thru 0x0132b2 (0x7481 bytes) 000307 1777573021.471858 read 0x01e5c5 thru 0x02a473 (0xbeaf bytes) 000309 1777573021.475389 trunc from 0x02d1bc to 0x037191 (0x9fd6 bytes) 000310 1777573021.480339 trunc from 0x037192 to 0x027201 (0xff90 bytes) 000311 1777573021.484586 read 0x0225fd thru 0x027201 (0x4c05 bytes) 000312 1777573021.485087 mapread 0x0081ed thru 0x01199b (0x97af bytes) 000313 1777573021.485839 read 0x00fb65 thru 0x015fd9 (0x6475 bytes) 000315 1777573021.486733 write 0x01a892 thru 0x01de17 (0x3586 bytes) 000317 1777573021.491078 mapread 0x016342 thru 0x01f5d2 (0x9291 bytes) 000318 1777573021.491911 mapread 0x01026d thru 0x012f0b (0x2c9f bytes) 000319 1777573021.492501 mapwrite 0x00dd88 thru 0x0159ed (0x7c66 bytes) 000322 1777573021.507425 read 0x0264d4 thru 0x027201 (0xd2e bytes) 000323 1777573021.507843 write 0x00ac36 thru 0x01aaaf (0xfe7a bytes) 000324 1777573021.508710 read 0x0252aa thru 0x027201 (0x1f58 bytes) 000325 1777573021.510796 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27202): Mirror 6: 0x1f27b0cb (stale) Mirror 7: 0x1f27b0cb (stale) Mirror 8: 0x515d0035 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27202): Mirror 6: 0x1f27b0cb (stale) Mirror 7: 0x1f27b0cb (stale) Mirror 8: 0x515d0035 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27202: skipped 000326 1777573021.609953 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x27202): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27202): Mirror 6: 0x1f27b0cb (stale) Mirror 7: 0x1f27b0cb (stale) Mirror 8: 0x515d0035 (stale) CRC-32 checksum value for chunk [0, 0x27202): Mirror 1: 0xb61fdf13 Mirror 9: 0xb61fdf13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27202: skipped 000327 1777573021.850531 mapwrite 0x027d66 thru 0x0295c3 (0x185e bytes) 000328 1777573021.896859 MIRROR_RESYNC 5 mirrors 000329 1777573022.154685 read 0x022d38 thru 0x0295c3 (0x688c bytes) 000331 1777573022.157334 mapread 0x011d95 thru 0x01f0ec (0xd358 bytes) 000332 1777573022.165175 write 0x002e3a thru 0x00b466 (0x862d bytes) 000333 1777573022.185298 write 0x035ccd thru 0x03ef2c (0x9260 bytes) 000334 1777573022.186560 mapread 0x0215a4 thru 0x026bc9 (0x5626 bytes) 000335 1777573022.190303 read 0x0280e6 thru 0x0334f0 (0xb40b bytes) 000337 1777573022.195494 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ef2d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef2d): Mirror 6: 0x2c25155b (stale) Mirror 7: 0x2c25155b (stale) Mirror 8: 0x2c25155b (stale) Mirror 9: 0x2c25155b (stale) CRC-32 checksum value for chunk [0, 0x3ef2d): Mirror 1: 0xe03aaf37 Mirror 10: 0xe03aaf37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef2d: skipped 000340 1777573022.438237 read 0x01442d thru 0x023bf2 (0xf7c6 bytes) 000341 1777573022.443835 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ef2d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef2d): Mirror 6: 0x2c25155b (stale) Mirror 7: 0x2c25155b (stale) Mirror 8: 0x2c25155b (stale) Mirror 9: 0x2c25155b (stale) CRC-32 checksum value for chunk [0, 0x3ef2d): Mirror 1: 0xe03aaf37 Mirror 10: 0xe03aaf37 Mirror 11: 0xe03aaf37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef2d: skipped 000342 1777573022.682090 trunc from 0x03ef2d to 0x01c404 (0x22b28 bytes) 000344 1777573022.708880 read 0x00786a thru 0x0174f4 (0xfc8b bytes) 000347 1777573022.712583 read 0x01468c thru 0x01c404 (0x7d79 bytes) 000349 1777573022.715672 mapwrite 0x0077b9 thru 0x01041e (0x8c66 bytes) 000352 1777573022.729548 trunc from 0x01c405 to 0x032d68 (0x16964 bytes) 000353 1777573022.735553 write 0x02706e thru 0x029db3 (0x2d46 bytes) 000357 1777573022.739517 mapwrite 0x0269c3 thru 0x032e58 (0xc496 bytes) 000359 1777573022.757788 read 0x028ac2 thru 0x02a461 (0x19a0 bytes) 000360 1777573022.758291 write 0x03586a thru 0x03b746 (0x5edd bytes) 000361 1777573022.761039 MIRROR_RESYNC 7 mirrors 000362 1777573023.045152 mapwrite 0x0252eb thru 0x02d668 (0x837e bytes) 000363 1777573023.097037 read 0x0053cb thru 0x006134 (0xd6a bytes) 000365 1777573023.100917 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b747): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b747): Mirror 6: 0x3eca8e2 (stale) Mirror 7: 0x3eca8e2 (stale) Mirror 8: 0x3eca8e2 (stale) Mirror 9: 0x3eca8e2 (stale) Mirror 10: 0x3eca8e2 (stale) Mirror 11: 0x3eca8e2 (stale) CRC-32 checksum value for chunk [0, 0x3b747): Mirror 1: 0x51272d28 Mirror 12: 0x51272d28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b747: skipped 000366 1777573023.346925 write 0x009fb9 thru 0x0158ae (0xb8f6 bytes) 000367 1777573023.370393 mapwrite 0x006278 thru 0x00ff07 (0x9c90 bytes) 000369 1777573023.397681 read 0x00e0e3 thru 0x017839 (0x9757 bytes) 000370 1777573023.400613 read 0x036478 thru 0x0386fe (0x2287 bytes) 000371 1777573023.404240 read 0x02d747 thru 0x03b746 (0xe000 bytes) 000372 1777573023.404715 mapwrite 0x002433 thru 0x00495a (0x2528 bytes) 000373 1777573023.417378 trunc from 0x03b747 to 0x01e358 (0x1d3ee bytes) 000374 1777573023.430546 trunc from 0x01e359 to 0x033831 (0x154d9 bytes) 000376 1777573023.438537 MIRROR_RESYNC 8 mirrors 000377 1777573023.766319 read 0x00f81c thru 0x019ac6 (0xa2ab bytes) 000378 1777573023.769232 read 0x02ecbc thru 0x033831 (0x4b76 bytes) 000379 1777573023.772712 read 0x01c3b7 thru 0x029934 (0xd57e bytes) 000380 1777573023.773140 trunc from 0x033832 to 0x032699 (0x1198 bytes) 000381 1777573023.796669 write 0x0321aa thru 0x037b61 (0x59b8 bytes) 000384 1777573023.801760 read 0x026645 thru 0x0284b8 (0x1e74 bytes) 000385 1777573023.802256 mapwrite 0x022585 thru 0x02929b (0x6d17 bytes) 000386 1777573023.818239 mapread 0x00ebc7 thru 0x0136b1 (0x4aeb bytes) 000387 1777573023.819073 trunc from 0x037b62 to 0x00f09f (0x28ac2 bytes) 000389 1777573023.825641 mapwrite 0x035a09 thru 0x03fca9 (0xa2a1 bytes) 000390 1777573023.847825 trunc from 0x03fcaa to 0x038079 (0x7c30 bytes) 000391 1777573023.853247 mapwrite 0x003cac thru 0x006307 (0x265c bytes) 000392 1777573023.877178 MIRROR_RESYNC 8 mirrors 000393 1777573024.217826 mapread 0x00b038 thru 0x0144c6 (0x948f bytes) 000396 1777573024.224549 read 0x0374da thru 0x038079 (0xba0 bytes) 000397 1777573024.226312 read 0x018285 thru 0x019923 (0x169f bytes) 000398 1777573024.229404 mapwrite 0x01c14a thru 0x025b2e (0x99e5 bytes) 000399 1777573024.291558 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3807a): Mirror 7: 0xe4cf305e (stale) Mirror 8: 0xe4cf305e (stale) Mirror 9: 0xe4cf305e (stale) Mirror 10: 0xe4cf305e (stale) Mirror 11: 0xe4cf305e (stale) Mirror 12: 0xe4cf305e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3807a): Mirror 7: 0xe4cf305e (stale) Mirror 8: 0xe4cf305e (stale) Mirror 9: 0xe4cf305e (stale) Mirror 10: 0xe4cf305e (stale) Mirror 11: 0xe4cf305e (stale) Mirror 12: 0xe4cf305e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3807a: skipped 000401 1777573024.430444 trunc from 0x03807a to 0x016093 (0x21fe6 bytes) 000402 1777573024.436321 mapread 0x001444 thru 0x00e632 (0xd1ef bytes) 000403 1777573024.443925 write 0x013f06 thru 0x01fb8f (0xbc8a bytes) 000404 1777573024.447401 trunc from 0x01fb90 to 0x00d543 (0x1264c bytes) 000406 1777573024.453706 write 0x02ddad thru 0x032da7 (0x4ffb bytes) 000409 1777573024.455296 write 0x01a9b1 thru 0x02862b (0xdc7b bytes) 000411 1777573024.460105 read 0x01bc84 thru 0x0232e6 (0x7663 bytes) 000412 1777573024.460608 write 0x005a81 thru 0x0117d3 (0xbd53 bytes) 000413 1777573024.462730 mapread 0x0212ca thru 0x027743 (0x647a bytes) 000414 1777573024.463566 trunc from 0x032da8 to 0x036b7f (0x3dd8 bytes) 000415 1777573024.468407 write 0x03e997 thru 0x03ffff (0x1669 bytes) 000417 1777573024.469982 write 0x036c18 thru 0x03ffff (0x93e8 bytes) 000419 1777573024.473413 mapwrite 0x032665 thru 0x03ffff (0xd99b bytes) 000420 1777573024.491601 mapwrite 0x0190dc thru 0x0252b5 (0xc1da bytes) 000421 1777573024.510040 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xeeba8201 (stale) Mirror 8: 0xeeba8201 (stale) Mirror 9: 0xeeba8201 (stale) Mirror 10: 0xeeba8201 (stale) Mirror 11: 0xeeba8201 (stale) Mirror 12: 0xeeba8201 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xddc1f81e Mirror 13: 0xddc1f81e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000425 1777573024.789988 mapwrite 0x020c70 thru 0x0225f0 (0x1981 bytes) 000426 1777573024.844094 mapread 0x03099b thru 0x03e09a (0xd700 bytes) 000427 1777573024.850165 mapwrite 0x010dc6 thru 0x019949 (0x8b84 bytes) 000428 1777573024.870860 mapwrite 0x01fa0f thru 0x0214a9 (0x1a9b bytes) 000429 1777573024.884258 trunc from 0x040000 to 0x00e938 (0x316c7 bytes) 000430 1777573024.903078 trunc from 0x00e939 to 0x030cb0 (0x22378 bytes) 000432 1777573024.907875 write 0x025641 thru 0x02f55a (0x9f1a bytes) 000434 1777573024.914298 MIRROR_RESYNC 8 mirrors 000435 1777573025.256389 read 0x01041a thru 0x0188f9 (0x84e0 bytes) 000436 1777573025.259180 write 0x02051a thru 0x02969e (0x9185 bytes) 000439 1777573025.279495 write 0x0337e2 thru 0x035ebc (0x26db bytes) 000443 1777573025.282064 trunc from 0x035ebd to 0x01e7b3 (0x17709 bytes) 000444 1777573025.290182 MIRROR_RESYNC 8 mirrors 000445 1777573025.618810 mapread 0x00da3e thru 0x011144 (0x3707 bytes) 000447 1777573025.624771 mapread 0x0000e9 thru 0x00aabf (0xa9d7 bytes) 000448 1777573025.627967 read 0x013be9 thru 0x016f04 (0x331c bytes) 000449 1777573025.628284 mapwrite 0x001d7b thru 0x008e57 (0x70dd bytes) 000451 1777573025.677134 trunc from 0x01e7b4 to 0x03b62d (0x1ce7a bytes) 000452 1777573025.684694 write 0x0016ba thru 0x00bac1 (0xa408 bytes) 000453 1777573025.686075 read 0x020ac1 thru 0x02f1f9 (0xe739 bytes) 000458 1777573025.692173 write 0x004a95 thru 0x01314a (0xe6b6 bytes) 000459 1777573025.693295 mapread 0x03919c thru 0x03b62d (0x2492 bytes) 000461 1777573025.696091 MIRROR_RESYNC 8 mirrors 000462 1777573026.014068 trunc from 0x03b62e to 0x00ed25 (0x2c908 bytes) 000463 1777573026.044289 MIRROR_RESYNC 8 mirrors 000464 1777573026.367292 mapwrite 0x03e3e2 thru 0x03ffff (0x1c1e bytes) 000465 1777573026.404634 read 0x03d542 thru 0x03ffff (0x2abe bytes) 000471 1777573026.409168 mapread 0x03eee1 thru 0x03ffff (0x111f bytes) 000474 1777573026.412281 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xc6c18aeb (stale) Mirror 8: 0xc6c18aeb (stale) Mirror 9: 0xc6c18aeb (stale) Mirror 10: 0xc6c18aeb (stale) Mirror 11: 0xc6c18aeb (stale) Mirror 12: 0xc6c18aeb (stale) Mirror 13: 0xc6c18aeb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd4950e00 Mirror 14: 0xd4950e00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000475 1777573026.696821 mapread 0x003d62 thru 0x00d674 (0x9913 bytes) 000476 1777573026.705931 mapwrite 0x022fe4 thru 0x02ee71 (0xbe8e bytes) 000479 1777573026.770545 write 0x02f796 thru 0x03e90f (0xf17a bytes) 000480 1777573026.773634 mapread 0x0121a2 thru 0x0176b7 (0x5516 bytes) 000481 1777573026.779871 MIRROR_RESYNC 9 mirrors 000482 1777573027.181240 write 0x033223 thru 0x03ffff (0xcddd bytes) 000483 1777573027.200586 read 0x0244e3 thru 0x02eb54 (0xa672 bytes) 000484 1777573027.205143 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x95d4a4c (stale) Mirror 8: 0x95d4a4c (stale) Mirror 9: 0x95d4a4c (stale) Mirror 10: 0x95d4a4c (stale) Mirror 11: 0x95d4a4c (stale) Mirror 13: 0x95d4a4c (stale) Mirror 14: 0x95d4a4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x95d4a4c (stale) Mirror 8: 0x95d4a4c (stale) Mirror 9: 0x95d4a4c (stale) Mirror 10: 0x95d4a4c (stale) Mirror 11: 0x95d4a4c (stale) Mirror 13: 0x95d4a4c (stale) Mirror 14: 0x95d4a4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000485 1777573027.346177 mapwrite 0x031c7b thru 0x03cde9 (0xb16f bytes) 000486 1777573027.373553 trunc from 0x040000 to 0x000a6f (0x3f590 bytes) 000487 1777573027.379174 trunc from 0x000a70 to 0x028236 (0x277c7 bytes) 000490 1777573027.384500 mapread 0x0104bf thru 0x020271 (0xfdb3 bytes) 000491 1777573027.390778 read 0x01b5ef thru 0x01c9bc (0x13ce bytes) 000492 1777573027.391434 read 0x01d5f5 thru 0x025570 (0x7f7c bytes) 000496 1777573027.396564 trunc from 0x028237 to 0x020e31 (0x7405 bytes) 000497 1777573027.402080 read 0x01c185 thru 0x01d100 (0xf7c bytes) 000500 1777573027.403321 write 0x0363fc thru 0x03ef79 (0x8b7e bytes) 000501 1777573027.404240 mapwrite 0x01f8a4 thru 0x01ffad (0x70a bytes) 000505 1777573027.418884 read 0x0062c3 thru 0x0109e8 (0xa726 bytes) 000507 1777573027.419902 read 0x00a621 thru 0x010404 (0x5de4 bytes) 000509 1777573027.420880 mapread 0x001c33 thru 0x00999b (0x7d69 bytes) 000510 1777573027.421721 trunc from 0x03ef7a to 0x0046d1 (0x3a8a8 bytes) 000511 1777573027.427924 write 0x007153 thru 0x015cc1 (0xeb6f bytes) 000514 1777573027.429910 mapread 0x0104d9 thru 0x015cc1 (0x57e9 bytes) 000516 1777573027.430994 mapread 0x007560 thru 0x00c0e8 (0x4b89 bytes) 000518 1777573027.432144 write 0x03f435 thru 0x03ffff (0xbcb bytes) 000521 1777573027.434284 mapread 0x01bb3d thru 0x028553 (0xca17 bytes) 000523 1777573027.439061 read 0x03d907 thru 0x03ffff (0x26f9 bytes) 000524 1777573027.439629 read 0x005146 thru 0x00c8f5 (0x77b0 bytes) 000525 1777573027.440044 mapwrite 0x028b83 thru 0x02ad82 (0x2200 bytes) 000526 1777573027.455896 mapread 0x016af6 thru 0x023c4e (0xd159 bytes) 000528 1777573027.457362 read 0x004f67 thru 0x00ddc1 (0x8e5b bytes) 000529 1777573027.459770 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x95d4a4c (stale) Mirror 8: 0x95d4a4c (stale) Mirror 9: 0x95d4a4c (stale) Mirror 10: 0x95d4a4c (stale) Mirror 11: 0x95d4a4c (stale) Mirror 13: 0x95d4a4c (stale) Mirror 14: 0x95d4a4c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9e63c928 Mirror 15: 0x9e63c928 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000534 1777573027.752566 trunc from 0x040000 to 0x01dec5 (0x2213a bytes) 000536 1777573027.782677 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1dec6): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dec6): Mirror 7: 0x3a490158 (stale) Mirror 8: 0x3a490158 (stale) Mirror 9: 0x3a490158 (stale) Mirror 10: 0x3a490158 (stale) Mirror 11: 0x3a490158 (stale) Mirror 13: 0x3a490158 (stale) Mirror 14: 0x3a490158 (stale) Mirror 15: 0x4a3281d5 (stale) CRC-32 checksum value for chunk [0, 0x1dec6): Mirror 1: 0x4a3281d5 Mirror 16: 0x4a3281d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dec6: skipped 000538 1777573028.066534 write 0x0122fa thru 0x01d827 (0xb52e bytes) 000539 1777573028.090408 write 0x0271d7 thru 0x02b4bf (0x42e9 bytes) 000540 1777573028.091193 mapread 0x025afd thru 0x02b2e5 (0x57e9 bytes) 000543 1777573028.094764 read 0x023706 thru 0x0299d3 (0x62ce bytes) 000544 1777573028.095160 read 0x005da0 thru 0x007a02 (0x1c63 bytes) 000545 1777573028.097853 write 0x0361d4 thru 0x03d3e9 (0x7216 bytes) 000548 1777573028.099374 mapread 0x01db5f thru 0x02d822 (0xfcc4 bytes) 000549 1777573028.103701 MIRROR_RESYNC 10 mirrors truncating to largest ever: 0x3fe5d 000550 1777573028.493728 trunc from 0x03d3ea to 0x03fe5c (0x2a73 bytes) 000554 1777573028.520888 mapread 0x0215fe thru 0x026691 (0x5094 bytes) 000555 1777573028.527375 write 0x008c71 thru 0x00d6a8 (0x4a38 bytes) 000556 1777573028.530714 read 0x01bfbc thru 0x01f19d (0x31e2 bytes) 000559 1777573028.533547 trunc from 0x03fe5d to 0x0155ac (0x2a8b0 bytes) 000560 1777573028.539153 read 0x001897 thru 0x00f1d6 (0xd940 bytes) 000561 1777573028.541056 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x155ad): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x155ad): Mirror 7: 0x402b7db4 (stale) Mirror 8: 0x402b7db4 (stale) Mirror 9: 0x402b7db4 (stale) Mirror 10: 0x402b7db4 (stale) Mirror 11: 0x402b7db4 (stale) Mirror 13: 0x402b7db4 (stale) Mirror 14: 0x402b7db4 (stale) Mirror 15: 0x402b7db4 (stale) Mirror 16: 0x402b7db4 (stale) CRC-32 checksum value for chunk [0, 0x155ad): Mirror 1: 0x49c30b56 Mirror 17: 0x49c30b56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x155ad: skipped 000563 1777573028.796299 MIRROR_RESYNC 11 mirrors 000564 1777573029.121479 write 0x03286e thru 0x0351ba (0x294d bytes) 000565 1777573029.138971 write 0x03e669 thru 0x03ffff (0x1997 bytes) 000566 1777573029.139895 trunc from 0x040000 to 0x00058a (0x3fa75 bytes) 000568 1777573029.150003 write 0x01c402 thru 0x025253 (0x8e52 bytes) 000569 1777573029.150961 mapwrite 0x023f5b thru 0x031591 (0xd637 bytes) 000571 1777573029.178087 mapread 0x024bba thru 0x02958e (0x49d5 bytes) 000572 1777573029.179126 trunc from 0x031592 to 0x0227d9 (0xedb8 bytes) 000574 1777573029.184612 write 0x00c5fd thru 0x01151d (0x4f21 bytes) 000575 1777573029.188057 mapread 0x00599c thru 0x00f856 (0x9ebb bytes) 000576 1777573029.189009 write 0x004541 thru 0x010754 (0xc214 bytes) 000579 1777573029.193248 mapread 0x001159 thru 0x005667 (0x450f bytes) 000583 1777573029.197229 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x227da): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x227da): Mirror 7: 0x2a1d7260 (stale) Mirror 8: 0x2a1d7260 (stale) Mirror 9: 0x2a1d7260 (stale) Mirror 10: 0x2a1d7260 (stale) Mirror 11: 0x2a1d7260 (stale) Mirror 13: 0x2a1d7260 (stale) Mirror 14: 0x2a1d7260 (stale) Mirror 15: 0x2a1d7260 (stale) Mirror 16: 0x2a1d7260 (stale) Mirror 17: 0x2a1d7260 (stale) CRC-32 checksum value for chunk [0, 0x227da): Mirror 1: 0xfa63aca Mirror 18: 0xfa63aca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x227da: skipped 000585 1777573029.460670 mapread 0x021984 thru 0x0227d9 (0xe56 bytes) 000586 1777573029.466806 trunc from 0x0227da to 0x02c1d9 (0x9a00 bytes) 000588 1777573029.497248 mapread 0x01176a thru 0x013dc7 (0x265e bytes) 000590 1777573029.501433 write 0x029b6a thru 0x0382b1 (0xe748 bytes) 000595 1777573029.508011 trunc from 0x0382b2 to 0x033bee (0x46c3 bytes) 000599 1777573029.516468 read 0x01ab1b thru 0x01c11c (0x1602 bytes) 000600 1777573029.519297 read 0x018ad4 thru 0x0266f4 (0xdc21 bytes) 000601 1777573029.522797 read 0x02a89a thru 0x033bee (0x9355 bytes) 000602 1777573029.524805 MIRROR_RESYNC 12 mirrors 000604 1777573029.962809 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) 000605 1777573030.095155 mapwrite 0x00af84 thru 0x01626a (0xb2e7 bytes) 000606 1777573030.155613 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x33bef): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33bef): Mirror 7: 0xa2f93c08 (stale) Mirror 8: 0xa2f93c08 (stale) Mirror 9: 0xa2f93c08 (stale) Mirror 10: 0xa2f93c08 (stale) Mirror 13: 0xa2f93c08 (stale) Mirror 14: 0xa2f93c08 (stale) Mirror 15: 0xa2f93c08 (stale) Mirror 16: 0xa2f93c08 (stale) Mirror 17: 0xa2f93c08 (stale) Mirror 18: 0xa2f93c08 (stale) CRC-32 checksum value for chunk [0, 0x33bef): Mirror 1: 0xa94e50ea Mirror 19: 0xa94e50ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33bef: skipped 000607 1777573030.464195 read 0x00c49a thru 0x00e008 (0x1b6f bytes) 000608 1777573030.467659 trunc from 0x033bef to 0x035908 (0x1d1a bytes) 000609 1777573030.492998 mapwrite 0x004dc3 thru 0x007214 (0x2452 bytes) 000610 1777573030.514217 mapwrite 0x010b29 thru 0x01a84c (0x9d24 bytes) 000611 1777573030.535264 trunc from 0x035909 to 0x02a2fc (0xb60c bytes) 000615 1777573030.542830 read 0x023079 thru 0x025e3e (0x2dc6 bytes) 000616 1777573030.546896 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2a2fd): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2fd): Mirror 7: 0xa2b2290e (stale) Mirror 8: 0xa2b2290e (stale) Mirror 9: 0xa2b2290e (stale) Mirror 10: 0xa2b2290e (stale) Mirror 13: 0xa2b2290e (stale) Mirror 14: 0xa2b2290e (stale) Mirror 15: 0xa2b2290e (stale) Mirror 16: 0xa2b2290e (stale) Mirror 17: 0xa2b2290e (stale) Mirror 18: 0xa2b2290e (stale) Mirror 19: 0x59cb79d2 (stale) CRC-32 checksum value for chunk [0, 0x2a2fd): Mirror 1: 0xd18d4980 Mirror 20: 0xd18d4980 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2fd: skipped 000617 1777573030.797818 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2a2fd): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2fd): Mirror 7: 0xa2b2290e (stale) Mirror 8: 0xa2b2290e (stale) Mirror 9: 0xa2b2290e (stale) Mirror 10: 0xa2b2290e (stale) Mirror 13: 0xa2b2290e (stale) Mirror 14: 0xa2b2290e (stale) Mirror 15: 0xa2b2290e (stale) Mirror 16: 0xa2b2290e (stale) Mirror 17: 0xa2b2290e (stale) Mirror 18: 0xa2b2290e (stale) Mirror 19: 0x59cb79d2 (stale) CRC-32 checksum value for chunk [0, 0x2a2fd): Mirror 1: 0xd18d4980 Mirror 20: 0xd18d4980 Mirror 21: 0xd18d4980 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2fd: skipped 000618 1777573031.064159 mapread 0x002709 thru 0x00a65e (0x7f56 bytes) 000621 1777573031.073582 read 0x01c4bc thru 0x02228d (0x5dd2 bytes) 000622 1777573031.076947 mapwrite 0x021f00 thru 0x027671 (0x5772 bytes) 000623 1777573031.126287 write 0x02bdf0 thru 0x031172 (0x5383 bytes) 000624 1777573031.128733 MIRROR_RESYNC 14 mirrors 000625 1777573031.574206 read 0x02d906 thru 0x031172 (0x386d bytes) 000626 1777573031.576615 write 0x013f81 thru 0x01af26 (0x6fa6 bytes) 000628 1777573031.596982 write 0x018ca3 thru 0x01b7bf (0x2b1d bytes) 000629 1777573031.600001 mapwrite 0x03dfc9 thru 0x03ffff (0x2037 bytes) 000631 1777573031.622188 write 0x034bd1 thru 0x03ffff (0xb42f bytes) 000632 1777573031.625286 mapwrite 0x00133a thru 0x007dd5 (0x6a9c bytes) 000633 1777573031.641198 trunc from 0x040000 to 0x01924e (0x26db1 bytes) 000634 1777573031.647407 read 0x009682 thru 0x016b25 (0xd4a4 bytes) 000636 1777573031.651069 mapread 0x008967 thru 0x0110e9 (0x8783 bytes) 000637 1777573031.652013 read 0x0105c2 thru 0x0107c6 (0x205 bytes) 000638 1777573031.654517 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1924f): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1924f): Mirror 7: 0x370d016d (stale) Mirror 8: 0x370d016d (stale) Mirror 9: 0x370d016d (stale) Mirror 10: 0x370d016d (stale) Mirror 13: 0x370d016d (stale) Mirror 14: 0x370d016d (stale) Mirror 15: 0x370d016d (stale) Mirror 16: 0x370d016d (stale) Mirror 17: 0x370d016d (stale) Mirror 18: 0x370d016d (stale) Mirror 19: 0x370d016d (stale) Mirror 20: 0x370d016d (stale) Mirror 21: 0x370d016d (stale) CRC-32 checksum value for chunk [0, 0x1924f): Mirror 1: 0xe3444a95 Mirror 22: 0xe3444a95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1924f: skipped 000639 1777573031.918143 read 0x0045a2 thru 0x00c143 (0x7ba2 bytes) 000640 1777573031.921987 mapread 0x000c67 thru 0x0011b7 (0x551 bytes) 000641 1777573031.928858 trunc from 0x01924f to 0x00ff39 (0x9315 bytes) 000642 1777573031.955728 write 0x009b3a thru 0x00d7a3 (0x3c6a bytes) 000643 1777573031.960914 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0xff3a): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xff3a): Mirror 7: 0xb699b828 (stale) Mirror 8: 0xb699b828 (stale) Mirror 9: 0xb699b828 (stale) Mirror 10: 0xb699b828 (stale) Mirror 13: 0xb699b828 (stale) Mirror 14: 0xb699b828 (stale) Mirror 15: 0xb699b828 (stale) Mirror 16: 0xb699b828 (stale) Mirror 17: 0xb699b828 (stale) Mirror 18: 0xb699b828 (stale) Mirror 19: 0xb699b828 (stale) Mirror 20: 0xb699b828 (stale) Mirror 21: 0xb699b828 (stale) Mirror 22: 0x92b491cd (stale) CRC-32 checksum value for chunk [0, 0xff3a): Mirror 1: 0x9fae0bba Mirror 23: 0x9fae0bba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xff3a: skipped 000644 1777573032.238197 mapwrite 0x016713 thru 0x017f0b (0x17f9 bytes) 000646 1777573032.275261 MIRROR_SPLIT mirror 6 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17f0c): Mirror 7: 0xd4c74244 (stale) Mirror 8: 0xd4c74244 (stale) Mirror 9: 0xd4c74244 (stale) Mirror 10: 0xd4c74244 (stale) Mirror 13: 0xd4c74244 (stale) Mirror 15: 0xd4c74244 (stale) Mirror 16: 0xd4c74244 (stale) Mirror 17: 0xd4c74244 (stale) Mirror 18: 0xd4c74244 (stale) Mirror 19: 0xd4c74244 (stale) Mirror 20: 0xd4c74244 (stale) Mirror 21: 0xd4c74244 (stale) Mirror 22: 0xf8be87e8 (stale) Mirror 23: 0xbd4d9535 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17f0c): Mirror 7: 0xd4c74244 (stale) Mirror 8: 0xd4c74244 (stale) Mirror 9: 0xd4c74244 (stale) Mirror 10: 0xd4c74244 (stale) Mirror 13: 0xd4c74244 (stale) Mirror 15: 0xd4c74244 (stale) Mirror 16: 0xd4c74244 (stale) Mirror 17: 0xd4c74244 (stale) Mirror 18: 0xd4c74244 (stale) Mirror 19: 0xd4c74244 (stale) Mirror 20: 0xd4c74244 (stale) Mirror 21: 0xd4c74244 (stale) Mirror 22: 0xf8be87e8 (stale) Mirror 23: 0xbd4d9535 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17f0c: skipped 000647 1777573032.441204 MIRROR_RESYNC 15 mirrors 000648 1777573032.876932 trunc from 0x017f0c to 0x00403b (0x13ed0 bytes) 000650 1777573032.898698 mapread 0x0032c8 thru 0x00403b (0xd74 bytes) 000651 1777573032.903221 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x403c): Mirror 8: 0xd59f3c37 (stale) Mirror 9: 0xd59f3c37 (stale) Mirror 10: 0xd59f3c37 (stale) Mirror 13: 0xd59f3c37 (stale) Mirror 15: 0xd59f3c37 (stale) Mirror 16: 0xd59f3c37 (stale) Mirror 17: 0xd59f3c37 (stale) Mirror 18: 0xd59f3c37 (stale) Mirror 19: 0xd59f3c37 (stale) Mirror 20: 0xd59f3c37 (stale) Mirror 21: 0xd59f3c37 (stale) Mirror 22: 0xd59f3c37 (stale) Mirror 23: 0xd59f3c37 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x403c): Mirror 8: 0xd59f3c37 (stale) Mirror 9: 0xd59f3c37 (stale) Mirror 10: 0xd59f3c37 (stale) Mirror 13: 0xd59f3c37 (stale) Mirror 15: 0xd59f3c37 (stale) Mirror 16: 0xd59f3c37 (stale) Mirror 17: 0xd59f3c37 (stale) Mirror 18: 0xd59f3c37 (stale) Mirror 19: 0xd59f3c37 (stale) Mirror 20: 0xd59f3c37 (stale) Mirror 21: 0xd59f3c37 (stale) Mirror 22: 0xd59f3c37 (stale) Mirror 23: 0xd59f3c37 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x403c: skipped 000653 1777573033.040584 trunc from 0x00403c to 0x0324b9 (0x2e47e bytes) 000657 1777573033.047272 MIRROR_RESYNC 14 mirrors 000659 1777573033.471928 mapwrite 0x02acc4 thru 0x034f9b (0xa2d8 bytes) 000660 1777573033.513523 read 0x00e0d8 thru 0x019643 (0xb56c bytes) 000661 1777573033.516654 mapwrite 0x03d7cc thru 0x03ffff (0x2834 bytes) 000663 1777573033.538103 MIRROR_RESYNC 14 mirrors 000665 1777573033.938354 mapwrite 0x0138ff thru 0x01b5bd (0x7cbf bytes) 000666 1777573033.989657 mapwrite 0x028280 thru 0x02d91e (0x569f bytes) 000667 1777573034.002726 read 0x01b2dd thru 0x021db9 (0x6add bytes) 000669 1777573034.005468 read 0x0294ff thru 0x038f25 (0xfa27 bytes) 000671 1777573034.008656 write 0x0091f6 thru 0x00ed87 (0x5b92 bytes) 000673 1777573034.009784 mapread 0x00b199 thru 0x015ea0 (0xad08 bytes) 000674 1777573034.010626 write 0x0040b0 thru 0x006c1c (0x2b6d bytes) 000675 1777573034.011344 mapwrite 0x02abb7 thru 0x02fdb4 (0x51fe bytes) 000676 1777573034.024949 mapread 0x0049f6 thru 0x00f200 (0xa80b bytes) 000678 1777573034.026080 mapwrite 0x0165a6 thru 0x017fb7 (0x1a12 bytes) 000679 1777573034.036386 mapread 0x018261 thru 0x01d5ab (0x534b bytes) 000681 1777573034.037466 write 0x002913 thru 0x00525b (0x2949 bytes) 000682 1777573034.038151 write 0x0001a3 thru 0x006f2b (0x6d89 bytes) 000685 1777573034.042407 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x9c75bd78 (stale) Mirror 9: 0x9c75bd78 (stale) Mirror 10: 0x9c75bd78 (stale) Mirror 13: 0x9c75bd78 (stale) Mirror 15: 0x9c75bd78 (stale) Mirror 16: 0x9c75bd78 (stale) Mirror 17: 0x9c75bd78 (stale) Mirror 18: 0x9c75bd78 (stale) Mirror 19: 0x9c75bd78 (stale) Mirror 20: 0x9c75bd78 (stale) Mirror 21: 0x9c75bd78 (stale) Mirror 23: 0x9c75bd78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x9c75bd78 (stale) Mirror 9: 0x9c75bd78 (stale) Mirror 10: 0x9c75bd78 (stale) Mirror 13: 0x9c75bd78 (stale) Mirror 15: 0x9c75bd78 (stale) Mirror 16: 0x9c75bd78 (stale) Mirror 17: 0x9c75bd78 (stale) Mirror 18: 0x9c75bd78 (stale) Mirror 19: 0x9c75bd78 (stale) Mirror 20: 0x9c75bd78 (stale) Mirror 21: 0x9c75bd78 (stale) Mirror 23: 0x9c75bd78 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000687 1777573034.215697 mapread 0x0075ea thru 0x0151a3 (0xdbba bytes) 000692 1777573034.226497 write 0x0334da thru 0x037e6f (0x4996 bytes) 000694 1777573034.233053 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9c75bd78 (stale) Mirror 10: 0x9c75bd78 (stale) Mirror 13: 0x9c75bd78 (stale) Mirror 15: 0x9c75bd78 (stale) Mirror 16: 0x9c75bd78 (stale) Mirror 17: 0x9c75bd78 (stale) Mirror 18: 0x9c75bd78 (stale) Mirror 19: 0x9c75bd78 (stale) Mirror 20: 0x9c75bd78 (stale) Mirror 21: 0x9c75bd78 (stale) Mirror 23: 0x9c75bd78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9c75bd78 (stale) Mirror 10: 0x9c75bd78 (stale) Mirror 13: 0x9c75bd78 (stale) Mirror 15: 0x9c75bd78 (stale) Mirror 16: 0x9c75bd78 (stale) Mirror 17: 0x9c75bd78 (stale) Mirror 18: 0x9c75bd78 (stale) Mirror 19: 0x9c75bd78 (stale) Mirror 20: 0x9c75bd78 (stale) Mirror 21: 0x9c75bd78 (stale) Mirror 23: 0x9c75bd78 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000696 1777573034.389032 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9c75bd78 (stale) Mirror 10: 0x9c75bd78 (stale) Mirror 13: 0x9c75bd78 (stale) Mirror 15: 0x9c75bd78 (stale) Mirror 16: 0x9c75bd78 (stale) Mirror 17: 0x9c75bd78 (stale) Mirror 18: 0x9c75bd78 (stale) Mirror 19: 0x9c75bd78 (stale) Mirror 20: 0x9c75bd78 (stale) Mirror 21: 0x9c75bd78 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x9c75bd78 (stale) Mirror 10: 0x9c75bd78 (stale) Mirror 13: 0x9c75bd78 (stale) Mirror 15: 0x9c75bd78 (stale) Mirror 16: 0x9c75bd78 (stale) Mirror 17: 0x9c75bd78 (stale) Mirror 18: 0x9c75bd78 (stale) Mirror 19: 0x9c75bd78 (stale) Mirror 20: 0x9c75bd78 (stale) Mirror 21: 0x9c75bd78 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000698 1777573034.527883 mapread 0x0274c5 thru 0x0284ed (0x1029 bytes) 000701 1777573034.531788 trunc from 0x040000 to 0x03e708 (0x18f7 bytes) 000702 1777573034.542566 MIRROR_RESYNC 11 mirrors 000704 1777573034.900307 mapread 0x033111 thru 0x03e708 (0xb5f8 bytes) 000705 1777573034.906664 read 0x00fa3c thru 0x015358 (0x591d bytes) 000707 1777573034.909059 write 0x00776e thru 0x00b9b4 (0x4247 bytes) 000708 1777573034.930358 read 0x010a73 thru 0x0179fc (0x6f8a bytes) 000710 1777573034.934252 mapread 0x02afed thru 0x037021 (0xc035 bytes) 000711 1777573034.936450 MIRROR_RESYNC 11 mirrors 000712 1777573035.292886 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) 000720 1777573035.430420 mapwrite 0x00f734 thru 0x0112bf (0x1b8c bytes) 000721 1777573035.482322 trunc from 0x03e709 to 0x02f20f (0xf4f9 bytes) 000723 1777573035.487585 write 0x00016b thru 0x006858 (0x66ee bytes) 000726 1777573035.491495 mapwrite 0x00194d thru 0x011206 (0xf8ba bytes) 000727 1777573035.505609 read 0x02c4cd thru 0x02f20f (0x2d43 bytes) 000731 1777573035.508657 mapwrite 0x039a64 thru 0x03ab92 (0x112f bytes) 000737 1777573035.527249 read 0x002b58 thru 0x00d011 (0xa4ba bytes) 000738 1777573035.527741 read 0x015757 thru 0x022c49 (0xd4f3 bytes) 000740 1777573035.531172 read 0x039f02 thru 0x03ab92 (0xc91 bytes) 000742 1777573035.531935 read 0x03995f thru 0x03ab92 (0x1234 bytes) 000744 1777573035.532712 mapread 0x014ac1 thru 0x01cfcf (0x850f bytes) 000745 1777573035.534925 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3ab93): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab93): Mirror 9: 0xad1b474f (stale) Mirror 10: 0xad1b474f (stale) Mirror 13: 0xad1b474f (stale) Mirror 15: 0xad1b474f (stale) Mirror 16: 0xad1b474f (stale) Mirror 17: 0xad1b474f (stale) Mirror 18: 0xad1b474f (stale) Mirror 20: 0xad1b474f (stale) Mirror 21: 0xad1b474f (stale) CRC-32 checksum value for chunk [0, 0x3ab93): Mirror 1: 0x38196957 Mirror 22: 0x38196957 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab93: skipped 000749 1777573035.777791 read 0x00fe76 thru 0x01114e (0x12d9 bytes) 000754 1777573035.785477 write 0x030dbf thru 0x033640 (0x2882 bytes) 000756 1777573035.809079 read 0x002582 thru 0x009b15 (0x7594 bytes) 000758 1777573035.816924 mapread 0x0112a4 thru 0x01c66c (0xb3c9 bytes) 000759 1777573035.817842 write 0x00de7c thru 0x01b4c0 (0xd645 bytes) 000762 1777573035.819573 mapread 0x0022f0 thru 0x00c209 (0x9f1a bytes) 000765 1777573035.821028 mapwrite 0x035218 thru 0x037553 (0x233c bytes) 000767 1777573035.836474 mapwrite 0x00f397 thru 0x01a0b5 (0xad1f bytes) 000768 1777573035.853472 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3ab93): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab93): Mirror 9: 0xad1b474f (stale) Mirror 10: 0xad1b474f (stale) Mirror 13: 0xad1b474f (stale) Mirror 15: 0xad1b474f (stale) Mirror 16: 0xad1b474f (stale) Mirror 17: 0xad1b474f (stale) Mirror 18: 0xad1b474f (stale) Mirror 20: 0xad1b474f (stale) Mirror 21: 0xad1b474f (stale) Mirror 22: 0x38196957 (stale) CRC-32 checksum value for chunk [0, 0x3ab93): Mirror 1: 0x629dbc00 Mirror 23: 0x629dbc00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab93: skipped 000770 1777573036.105702 mapwrite 0x009c82 thru 0x011f53 (0x82d2 bytes) 000772 1777573036.162067 mapwrite 0x038052 thru 0x03ffff (0x7fae bytes) 000774 1777573036.186679 mapwrite 0x00e5c6 thru 0x01d90d (0xf348 bytes) 000775 1777573036.209697 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3de21472 (stale) Mirror 10: 0x3de21472 (stale) Mirror 13: 0x3de21472 (stale) Mirror 15: 0x3de21472 (stale) Mirror 16: 0x3de21472 (stale) Mirror 17: 0x3de21472 (stale) Mirror 18: 0x3de21472 (stale) Mirror 20: 0x3de21472 (stale) Mirror 21: 0x3de21472 (stale) Mirror 22: 0x22b65081 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3de21472 (stale) Mirror 10: 0x3de21472 (stale) Mirror 13: 0x3de21472 (stale) Mirror 15: 0x3de21472 (stale) Mirror 16: 0x3de21472 (stale) Mirror 17: 0x3de21472 (stale) Mirror 18: 0x3de21472 (stale) Mirror 20: 0x3de21472 (stale) Mirror 21: 0x3de21472 (stale) Mirror 22: 0x22b65081 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000777 1777573036.371812 read 0x03b5fb thru 0x03d523 (0x1f29 bytes) 000780 1777573036.375370 mapread 0x03116b thru 0x03f6ae (0xe544 bytes) 000781 1777573036.384068 mapwrite 0x023373 thru 0x02389c (0x52a bytes) 000782 1777573036.396799 mapwrite 0x009d1d thru 0x00bac3 (0x1da7 bytes) 000783 1777573036.409373 mapwrite 0x0042bd thru 0x00d394 (0x90d8 bytes) 000787 1777573036.422399 trunc from 0x040000 to 0x03c1b0 (0x3e4f bytes) 000788 1777573036.427154 mapread 0x00f9d5 thru 0x01403e (0x466a bytes) 000789 1777573036.430544 trunc from 0x03c1b1 to 0x01b2bc (0x20ef4 bytes) 000790 1777573036.435081 mapwrite 0x034bd9 thru 0x03ffff (0xb427 bytes) 000793 1777573036.459697 mapwrite 0x02f881 thru 0x0362b3 (0x6a33 bytes) 000794 1777573036.474075 mapwrite 0x001e7b thru 0x00b23c (0x93c2 bytes) 000795 1777573036.489807 write 0x03e793 thru 0x03ffff (0x186d bytes) 000796 1777573036.490465 mapread 0x01aa41 thru 0x01dfbd (0x357d bytes) 000797 1777573036.493788 trunc from 0x040000 to 0x033492 (0xcb6d bytes) 000798 1777573036.498342 read 0x000311 thru 0x0025c3 (0x22b3 bytes) 000800 1777573036.501280 write 0x006f44 thru 0x00bb5e (0x4c1b bytes) 000801 1777573036.501941 read 0x018cc6 thru 0x0240b5 (0xb3f0 bytes) 000803 1777573036.502660 trunc from 0x033493 to 0x027ecc (0xb5c6 bytes) 000807 1777573036.509175 trunc from 0x027ecd to 0x00a195 (0x1dd37 bytes) 000808 1777573036.514513 trunc from 0x00a196 to 0x0036f0 (0x6aa5 bytes) 000810 1777573036.520326 trunc from 0x0036f1 to 0x010f7f (0xd88f bytes) 000811 1777573036.525592 read 0x00b31b thru 0x00b36b (0x51 bytes) 000815 1777573036.530615 read 0x00deb8 thru 0x010f7f (0x30c8 bytes) 000816 1777573036.531071 mapwrite 0x0294b6 thru 0x02e8e9 (0x5434 bytes) 000817 1777573036.552504 mapread 0x018d31 thru 0x01f0d6 (0x63a6 bytes) 000818 1777573036.555219 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2e8ea): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e8ea): Mirror 9: 0x4da36c32 (stale) Mirror 10: 0x4da36c32 (stale) Mirror 13: 0x4da36c32 (stale) Mirror 15: 0x4da36c32 (stale) Mirror 16: 0x4da36c32 (stale) Mirror 17: 0x4da36c32 (stale) Mirror 18: 0x4da36c32 (stale) Mirror 20: 0x4da36c32 (stale) Mirror 21: 0x4da36c32 (stale) Mirror 22: 0xe12cdcc8 (stale) CRC-32 checksum value for chunk [0, 0x2e8ea): Mirror 1: 0xda2a5c32 Mirror 23: 0xda2a5c32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e8ea: skipped 000819 1777573036.813093 mapwrite 0x001068 thru 0x008e27 (0x7dc0 bytes) 000820 1777573036.871309 mapread 0x000a02 thru 0x001973 (0xf72 bytes) 000823 1777573036.874770 trunc from 0x02e8ea to 0x02f037 (0x74e bytes) 000824 1777573036.884137 write 0x016704 thru 0x018786 (0x2083 bytes) 000827 1777573036.887969 mapread 0x0182da thru 0x023879 (0xb5a0 bytes) 000829 1777573036.892218 write 0x02c7c7 thru 0x0304c3 (0x3cfd bytes) 000830 1777573036.896997 MIRROR_RESYNC 12 mirrors 000831 1777573037.302962 write 0x010ba6 thru 0x01d990 (0xcdeb bytes) 000832 1777573037.323443 mapread 0x00d1f7 thru 0x00fc6b (0x2a75 bytes) 000833 1777573037.326471 mapwrite 0x027403 thru 0x030b12 (0x9710 bytes) 000834 1777573037.350945 read 0x01109e thru 0x018816 (0x7779 bytes) 000836 1777573037.352244 trunc from 0x030b13 to 0x029109 (0x7a09 bytes) 000837 1777573037.356668 mapread 0x00b92f thru 0x010640 (0x4d12 bytes) 000839 1777573037.357694 mapwrite 0x006efd thru 0x015b6d (0xec71 bytes) 000842 1777573037.374534 write 0x00ae09 thru 0x00fa97 (0x4c8f bytes) 000843 1777573037.375430 read 0x00a38a thru 0x012df5 (0x8a6c bytes) 000845 1777573037.378212 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2910a): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2910a): Mirror 9: 0x5b42aae0 (stale) Mirror 10: 0x5b42aae0 (stale) Mirror 13: 0x5b42aae0 (stale) Mirror 15: 0x5b42aae0 (stale) Mirror 16: 0x5b42aae0 (stale) Mirror 17: 0x5b42aae0 (stale) Mirror 18: 0x5b42aae0 (stale) Mirror 20: 0x5b42aae0 (stale) Mirror 21: 0x5b42aae0 (stale) Mirror 22: 0x5b42aae0 (stale) Mirror 23: 0x5b42aae0 (stale) CRC-32 checksum value for chunk [0, 0x2910a): Mirror 1: 0x6fad5822 Mirror 24: 0x6fad5822 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2910a: skipped 000846 1777573037.642193 mapwrite 0x028f85 thru 0x029e53 (0xecf bytes) 000847 1777573037.686556 mapread 0x018a4b thru 0x01e130 (0x56e6 bytes) 000848 1777573037.690616 write 0x030bed thru 0x03ffff (0xf413 bytes) 000849 1777573037.691992 mapread 0x037cc9 thru 0x03ffff (0x8337 bytes) 000850 1777573037.692859 trunc from 0x040000 to 0x03049c (0xfb63 bytes) 000853 1777573037.700797 mapread 0x005a7a thru 0x006409 (0x990 bytes) 000854 1777573037.704446 mapread 0x02e7a5 thru 0x03049c (0x1cf8 bytes) 000855 1777573037.707585 trunc from 0x03049d to 0x031a41 (0x15a5 bytes) 000859 1777573037.714134 read 0x02d3fb thru 0x031a41 (0x4647 bytes) 000860 1777573037.716354 mapread 0x013458 thru 0x0214ac (0xe055 bytes) 000862 1777573037.719917 read 0x02b486 thru 0x02d258 (0x1dd3 bytes) 000866 1777573037.721700 write 0x029b27 thru 0x03873a (0xec14 bytes) 000867 1777573037.723174 trunc from 0x03873b to 0x00b471 (0x2d2c9 bytes) 000872 1777573037.731215 mapread 0x0084f6 thru 0x00b471 (0x2f7c bytes) 000873 1777573037.732127 trunc from 0x00b472 to 0x017ae9 (0xc678 bytes) 000874 1777573037.736833 mapread 0x0012b7 thru 0x0016b9 (0x403 bytes) 000875 1777573037.737460 write 0x020642 thru 0x02ebb0 (0xe56f bytes) 000878 1777573037.739741 mapwrite 0x03f28f thru 0x03ffff (0xd71 bytes) 000880 1777573037.762935 write 0x0353a7 thru 0x03a3fe (0x5058 bytes) 000881 1777573037.764114 mapread 0x00dd1b thru 0x01b0ff (0xd3e5 bytes) 000882 1777573037.765380 trunc from 0x040000 to 0x0312ca (0xed35 bytes) 000883 1777573037.770904 trunc from 0x0312cb to 0x00704f (0x2a27b bytes) 000885 1777573037.775649 write 0x01b14b thru 0x021e72 (0x6d28 bytes) 000887 1777573037.777481 trunc from 0x021e73 to 0x003e7a (0x1dff8 bytes) 000890 1777573037.783107 trunc from 0x003e7b to 0x022f4a (0x1f0d0 bytes) 000892 1777573037.787779 mapwrite 0x012412 thru 0x014986 (0x2575 bytes) 000893 1777573037.803286 trunc from 0x022f4b to 0x00f862 (0x136e8 bytes) 000894 1777573037.808440 mapread 0x000f03 thru 0x007e2a (0x6f28 bytes) 000895 1777573037.813684 trunc from 0x00f863 to 0x020dc5 (0x11563 bytes) 000896 1777573037.818405 trunc from 0x020dc6 to 0x0243ea (0x3625 bytes) 000898 1777573037.823012 mapwrite 0x000e4d thru 0x007b22 (0x6cd6 bytes) 000899 1777573037.836436 read 0x00ccbe thru 0x00d102 (0x445 bytes) 000900 1777573037.839780 mapread 0x0161ca thru 0x01f643 (0x947a bytes) 000901 1777573037.842725 MIRROR_RESYNC 13 mirrors 000903 1777573038.256207 mapread 0x00a649 thru 0x01a066 (0xfa1e bytes) 000905 1777573038.264335 MIRROR_RESYNC 13 mirrors 000907 1777573038.441961 write 0x0278d9 thru 0x02dc4a (0x6372 bytes) 000909 1777573038.461592 mapread 0x0299de thru 0x02dc4a (0x426d bytes) 000911 1777573038.464291 MIRROR_RESYNC 13 mirrors 000913 1777573038.853645 MIRROR_RESYNC 13 mirrors 000915 1777573039.044478 mapwrite 0x01d3f3 thru 0x024304 (0x6f12 bytes) 000916 1777573039.096779 write 0x021922 thru 0x02276e (0xe4d bytes) 000917 1777573039.097649 read 0x01d060 thru 0x02b416 (0xe3b7 bytes) 000918 1777573039.100350 trunc from 0x02dc4b to 0x01eea9 (0xeda1 bytes) 000920 1777573039.110561 write 0x006dec thru 0x01495e (0xdb73 bytes) 000922 1777573039.114954 write 0x02610b thru 0x029740 (0x3636 bytes) 000923 1777573039.117041 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x29741): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29741): Mirror 9: 0x7bb75fa5 (stale) Mirror 10: 0x7bb75fa5 (stale) Mirror 13: 0x7bb75fa5 (stale) Mirror 15: 0x7bb75fa5 (stale) Mirror 16: 0x7bb75fa5 (stale) Mirror 17: 0x7bb75fa5 (stale) Mirror 18: 0x7bb75fa5 (stale) Mirror 20: 0x7bb75fa5 (stale) Mirror 21: 0x7bb75fa5 (stale) Mirror 22: 0x7bb75fa5 (stale) Mirror 23: 0x7bb75fa5 (stale) Mirror 24: 0x7bb75fa5 (stale) CRC-32 checksum value for chunk [0, 0x29741): Mirror 1: 0xfb6311f7 Mirror 25: 0xfb6311f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29741: skipped 000924 1777573039.400526 mapwrite 0x00efbd thru 0x01971f (0xa763 bytes) 000927 1777573039.458118 read 0x01c6c3 thru 0x029740 (0xd07e bytes) 000928 1777573039.461221 trunc from 0x029741 to 0x01fb09 (0x9c37 bytes) 000929 1777573039.470534 mapwrite 0x00f147 thru 0x00f679 (0x533 bytes) 000931 1777573039.482267 mapwrite 0x023781 thru 0x0306fd (0xcf7d bytes) 000932 1777573039.506580 trunc from 0x0306fe to 0x00f40a (0x212f3 bytes) 000933 1777573039.511237 mapwrite 0x004e4d thru 0x006ad4 (0x1c88 bytes) 000934 1777573039.526065 read 0x002cd3 thru 0x00b9b8 (0x8ce6 bytes) 000936 1777573039.527147 read 0x00c879 thru 0x00f40a (0x2b92 bytes) 000937 1777573039.527797 mapread 0x0088b5 thru 0x00f40a (0x6b56 bytes) 000938 1777573039.528883 write 0x0333c6 thru 0x033dd8 (0xa13 bytes) 000939 1777573039.529939 mapwrite 0x01c994 thru 0x0293d8 (0xca45 bytes) 000940 1777573039.547780 write 0x011560 thru 0x012dd8 (0x1879 bytes) 000944 1777573039.549859 read 0x0196d3 thru 0x01cc35 (0x3563 bytes) 000947 1777573039.553130 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33dd9): Mirror 9: 0xec116218 (stale) Mirror 10: 0xec116218 (stale) Mirror 13: 0xec116218 (stale) Mirror 15: 0xec116218 (stale) Mirror 16: 0xec116218 (stale) Mirror 17: 0xec116218 (stale) Mirror 18: 0xec116218 (stale) Mirror 20: 0xec116218 (stale) Mirror 21: 0xec116218 (stale) Mirror 22: 0xec116218 (stale) Mirror 23: 0xec116218 (stale) Mirror 24: 0xec116218 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33dd9): Mirror 9: 0xec116218 (stale) Mirror 10: 0xec116218 (stale) Mirror 13: 0xec116218 (stale) Mirror 15: 0xec116218 (stale) Mirror 16: 0xec116218 (stale) Mirror 17: 0xec116218 (stale) Mirror 18: 0xec116218 (stale) Mirror 20: 0xec116218 (stale) Mirror 21: 0xec116218 (stale) Mirror 22: 0xec116218 (stale) Mirror 23: 0xec116218 (stale) Mirror 24: 0xec116218 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33dd9: skipped 000950 1777573039.718219 trunc from 0x033dd9 to 0x01549c (0x1e93c bytes) 000952 1777573039.723530 mapwrite 0x01c46e thru 0x023624 (0x71b7 bytes) 000954 1777573039.746301 mapread 0x01bafb thru 0x01d7ed (0x1cf3 bytes) 000957 1777573039.747876 read 0x0037b4 thru 0x01135c (0xdba9 bytes) 000958 1777573039.751348 mapwrite 0x035c35 thru 0x03c762 (0x6b2e bytes) 000959 1777573039.775606 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x661:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c763): Mirror 9: 0x958fcaf0 (stale) Mirror 10: 0x958fcaf0 (stale) Mirror 13: 0x958fcaf0 (stale) Mirror 15: 0x958fcaf0 (stale) Mirror 16: 0x958fcaf0 (stale) Mirror 18: 0x958fcaf0 (stale) Mirror 20: 0x958fcaf0 (stale) Mirror 21: 0x958fcaf0 (stale) Mirror 22: 0x958fcaf0 (stale) Mirror 23: 0x958fcaf0 (stale) Mirror 24: 0x958fcaf0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c763): Mirror 9: 0x958fcaf0 (stale) Mirror 10: 0x958fcaf0 (stale) Mirror 13: 0x958fcaf0 (stale) Mirror 15: 0x958fcaf0 (stale) Mirror 16: 0x958fcaf0 (stale) Mirror 18: 0x958fcaf0 (stale) Mirror 20: 0x958fcaf0 (stale) Mirror 21: 0x958fcaf0 (stale) Mirror 22: 0x958fcaf0 (stale) Mirror 23: 0x958fcaf0 (stale) Mirror 24: 0x958fcaf0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c763: skipped 000960 1777573039.935538 read 0x02eb06 thru 0x03bba4 (0xd09f bytes) 000961 1777573039.938075 read 0x0397f6 thru 0x03c762 (0x2f6d bytes) 000963 1777573039.941586 write 0x02034f thru 0x025e57 (0x5b09 bytes) 000966 1777573039.943073 trunc from 0x03c763 to 0x020cab (0x1bab7 bytes) 000969 1777573039.949433 MIRROR_RESYNC 12 mirrors 000970 1777573040.329196 mapwrite 0x02bef2 thru 0x02ca34 (0xb43 bytes) 000971 1777573040.365295 read 0x01fedb thru 0x023a4d (0x3b73 bytes) 000973 1777573040.368975 mapwrite 0x011e01 thru 0x017f8b (0x618b bytes) 000975 1777573040.384914 read 0x020d0b thru 0x0275aa (0x68a0 bytes) 000976 1777573040.388338 mapwrite 0x01e168 thru 0x0243b5 (0x624e bytes) 000978 1777573040.401649 read 0x006bb5 thru 0x011176 (0xa5c2 bytes) 000979 1777573040.402140 read 0x0118bc thru 0x01f154 (0xd899 bytes) 000980 1777573040.402583 read 0x0299b2 thru 0x02a178 (0x7c7 bytes) 000983 1777573040.403675 mapread 0x012695 thru 0x01950e (0x6e7a bytes) 000985 1777573040.404645 mapwrite 0x001086 thru 0x00c861 (0xb7dc bytes) 000987 1777573040.417726 write 0x0075c6 thru 0x00b2aa (0x3ce5 bytes) 000989 1777573040.419090 mapwrite 0x02b953 thru 0x033144 (0x77f2 bytes) 000990 1777573040.442380 trunc from 0x033145 to 0x01152c (0x21c18 bytes) 000991 1777573040.447049 mapread 0x00cc39 thru 0x01152c (0x48f4 bytes) 000992 1777573040.447808 mapread 0x0086d0 thru 0x00ef8e (0x68bf bytes) 000994 1777573040.448837 mapread 0x005d95 thru 0x01152c (0xb798 bytes) 000995 1777573040.449775 mapread 0x0028c6 thru 0x005637 (0x2d72 bytes) 000999 1777573040.451998 read 0x002900 thru 0x00b2cb (0x89cc bytes) 001000 1777573040.452530 trunc from 0x01152d to 0x022935 (0x11409 bytes) All operations completed A-OK! PASS 70a (26s) == sanity-flr test 71: check mirror extend parent fid ==== 14:17:22 (1777573042) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 14:17:23 (1777573043) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0706364 s, 29.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa00:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441782 s, 23.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa00:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x9ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa00:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9e3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa01:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (2s) == sanity-flr test 200a: stress test ===================== 14:17:25 (1777573045) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre2 Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1186240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4053344 .. done Extending file size to 7563648 .. Extending file size to 6398912 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6480448 .. failed Extending file size to 8848480 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7932448 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8826240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8916800 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9256160 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9098048 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Waiting 60037 60038 60040 60042 60044 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) 192.168.203.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg342-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (66s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:18:31 (1777573111) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre2 Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5478752 .. Extending file size to 5214112 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7243712 .. Extending file size to 6926016 .. Extending file size to 8972448 .. Extending file size to 6930048 .. Extending file size to 8864544 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9312224 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9356992 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 63165 63166 63168 63170 63172 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) 192.168.203.142@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg342-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:19:36 (1777573176) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3712 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210176 3712 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0384693 s, 54.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0796358 s, 26.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5e845efd (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) PASS 200c (6s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:19:42 (1777573182) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0285862 s, 73.4 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 14:19:45 (1777573185) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9fa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9fa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9fb:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0xa16:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:19:47 (1777573187) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0574196 s, 36.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa17:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa18:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:19:50 (1777573190) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0532332 s, 39.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa19:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa19:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1a:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:19:52 (1777573192) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0403637 s, 52.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x9ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1c:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:20:17 (1777573217) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg342-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027911 s, 37.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa00:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa00:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa01:0x0] } PASS 204d (42s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:20:59 (1777573259) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg342-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa20:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.049453 s, 42.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa20:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa02:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa1f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa20:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa02:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:21:46 (1777573306) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg342-server mds-ost sync done. Creating new pool test_204f oleg342-server: Pool lustre.test_204f created Adding targets to pool oleg342-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg342-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0708281 s, 29.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa21:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa03:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xa21:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa03:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg342-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg342-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg342-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (60s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:22:46 (1777573366) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:22:49 (1777573369) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:22:52 (1777573372) oleg342-server: Pool lustre.test_206 created Adding targets to pool oleg342-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg342-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg342-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg342-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg342-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:23:07 (1777573387) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294162 s, 35.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143322 s, 73.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x674a11ac Mirror 3: 0x674a11ac PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:23:11 (1777573391) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:23:12 (1777573392) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:23:14 (1777573394) Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0454788 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x6b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2080000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2080000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2080000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xa70000 Read this (4 bytes): BBBB 192.168.203.142@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg342-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:23:32 (1777573412) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00729126 s, 1.1 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:23:35 (1777573415) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215002 s, 48.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6c4 lmm_fid: [0x200000404:0x6c4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x6c4 lmm_fid: [0x200000404:0x6c4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xa0d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835652 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 14:23:35.000000000 -0400 Modify: 2026-04-30 14:23:35.000000000 -0400 Change: 2026-04-30 14:23:36.000000000 -0400 Birth: 2026-04-30 14:23:35.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:23:38 (1777573418) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:23:40 (1777573420) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.19508 s, 210 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0223446 s, 183 kB/s size after second write -rw-r--r-- 1 root root 4096 Apr 30 14:23 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x6c6:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Apr 30 14:23 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:23:43 (1777573423) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:23:46 (1777573426) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 830 sec ============ 14:23:49 (1777573429) === sanity-flr: start cleanup 14:23:50 (1777573430) === === sanity-flr: finish cleanup 14:23:52 (1777573432) ===