-----============= acceptance-small: sanity-flr ============----- Sat Aug 22 04:38:53 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg423-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 04:39:12 (1787387952) === oleg423-client.virtnet: executing check_config_client /mnt/lustre oleg423-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg423-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b58869a5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b58869a5000.idle_timeout=debug disable quota as required oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 04:39:46 (1787387986) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7590] == sanity-flr test 0a: lfs mirror create with -N option == 04:39:49 (1787387989) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:40:09 (1787388009) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:40:13 (1787388013) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:40:16 (1787388016) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:40:42 (1787388042) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:40:45 (1787388045) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 04:40:49 (1787388049) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b58869a5000.stats=clear osc.lustre-OST0001-osc-ffff9b58869a5000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:40:57 (1787388057) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:41:09 (1787388069) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:41:18 (1787388078) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:41:22 (1787388082) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:41:29 (1787388089) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 04:41:39 (1787388099) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:41:46 (1787388106) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:41:49 (1787388109) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:41:56 (1787388116) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:42:06 (1787388126) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:42:13 (1787388133) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.43971 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119341 s, 8.8 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:42:22 (1787388142) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.174124 s, 30.1 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:42:31 (1787388151) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:42:41 (1787388161) Stopping /mnt/lustre-ost1 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b58869a5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b58869a5000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg423-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b58869a5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b58869a5000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg423-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:43:53 (1787388233) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.386914 s, 19.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b58869a5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b58869a5000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg423-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:44:34 (1787388274) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b58869a5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b58869a5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg423-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b58869a5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b58869a5000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg423-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:45:50 (1787388350) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.88859 s, 10.7 MB/s -rw-r--r-- 1 root root 52428800 Aug 22 04:45 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.908s user 0m0.000s sys 0m0.181s Stopping /mnt/lustre-ost1 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b58869a5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b58869a5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.221s user 0m0.000s sys 0m0.182s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg423-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg423-server oleg423-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b58869a5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b58869a5000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.351s user 0m0.002s sys 0m0.192s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg423-server: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg423-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg423-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b58869a5000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b58869a5000.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:47:24 (1787388444) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:47:27 (1787388447) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:47:30 (1787388450) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:47:33 (1787388453) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 04:47:41 (1787388461) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.650834 s, 19.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:47:55 (1787388475) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331946 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258102 s, 40.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:48:02 (1787388482) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:48:11 (1787388491) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108649 s, 942 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00139894 s, 732 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00144657 s, 708 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00176103 s, 581 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 04:48:22 (1787388502) create test files with size 15746788 .. cc567835dc7acf4b0760302301e63a4e - 6fae088937448cc5fbd299a4ae8868e4 - 9ac42e41501b9bd43bdb021dbaee6080 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg423-client.virtnet /mnt/lustre (opts:) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 04:48:48 (1787388528) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:49:05 (1787388545) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:49:12 (1787388552) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167692 s, 6.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046851 s, 22.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 04:49:23 (1787388563) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11669 s, 18.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.361835 s, 11.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.384405 s, 5.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 04:49:43 (1787388583) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 04:49:46 (1787388586) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:49:50 (1787388590) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335606 s, 31.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 04:49:58 (1787388598) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:50:01 (1787388601) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:50:11 (1787388611) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.325638 s, 32.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg423-client.virtnet /mnt/lustre (opts:) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:50:23 (1787388623) llite.lustre-ffff9b5887f87000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.336348 s, 31.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg423-client.virtnet /mnt/lustre (opts:) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b5890643000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:50:38 (1787388638) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:51:18 (1787388678) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:52:00 (1787388720) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 04:52:04 (1787388724) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.330069 s, 9.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.191293 s, 16.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x12346699 (nosync,stale) Mirror 2: 0x12346699 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 45d0be6db8804de0aeafd0e23ae2b2ea - mirror 1 checksum: 51489305feddf05804efb0e7cc20fcd0 - mirror 2 checksum: 45d0be6db8804de0aeafd0e23ae2b2ea - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T04:52:08-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T04:52:08-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 45d0be6db8804de0aeafd0e23ae2b2ea - mirror 1 checksum: 51489305feddf05804efb0e7cc20fcd0 - mirror 2 checksum: 51489305feddf05804efb0e7cc20fcd0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T04:52:08-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T04:52:08-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:52:17 (1787388737) Failing mds1 on oleg423-server Stopping /mnt/lustre-mds1 (opts:) on oleg423-server 04:52:22 (1787388742) shut down facet: mds1 facet_host: oleg423-server facet_failover_host: oleg423-server Failover mds1 to oleg423-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 04:52:44 (1787388764) targets are mounted 04:52:44 (1787388764) facet_failover done oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272099 s, 38.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } PASS 50A (42s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:52:59 (1787388779) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:53:03 (1787388783) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:53:07 (1787388787) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00625942 s, 2.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:53:15 (1787388795) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:53:19 (1787388799) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:53:24 (1787388804) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:53:51 (1787388831) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:54:23 (1787388863) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:54:48 (1787388888) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:55:14 (1787388914) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 04:55:22 (1787388922) mirror create pid 53753 mirror split pid 53755 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53753 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53755 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 04:56:31 (1787388991) Seed set to 8993 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x3ca43 000004 1787388994.256975 trunc from 00000000 to 0x03ca42 (0x3ca43 bytes) 000005 1787388994.292026 read 0x0068fc thru 0x00d555 (0x6c5a bytes) 000009 1787388994.307049 mapwrite 0x0220a9 thru 0x02640b (0x4363 bytes) 000010 1787388994.414744 trunc from 0x03ca43 to 0x01f55c (0x1d4e6 bytes) 000011 1787388994.439362 read 0x00f9a8 thru 0x01efa3 (0xf5fc bytes) 000012 1787388994.467717 read 0x001002 thru 0x0104f0 (0xf4ef bytes) 000014 1787388994.475273 write 0x039452 thru 0x03ea3e (0x55ed bytes) 000015 1787388994.480727 trunc from 0x03ea3f to 0x01bcc4 (0x22d7a bytes) 000018 1787388994.521674 mapwrite 0x0334bc thru 0x0349f1 (0x1536 bytes) 000019 1787388994.687418 MIRROR_EXTEND to 2 mirrors 000020 1787388995.586122 mapread 0x02c51f thru 0x030bdd (0x46bf bytes) 000021 1787388995.620918 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) 000024 1787388995.779078 mapwrite 0x01336b thru 0x01f904 (0xc59a bytes) 000029 1787388996.077131 trunc from 0x0349f2 to 0x00b159 (0x29898 bytes) 000033 1787388996.128029 mapwrite 0x0083d5 thru 0x016aba (0xe6e6 bytes) 000034 1787388996.382461 mapwrite 0x014361 thru 0x01b3cd (0x706d bytes) 000035 1787388996.611038 write 0x022d8c thru 0x025dc3 (0x3038 bytes) 000039 1787388996.621273 mapwrite 0x0101c8 thru 0x01b6f9 (0xb532 bytes) 000040 1787388996.738659 MIRROR_EXTEND to 2 mirrors 000041 1787388997.773701 mapwrite 0x0290d3 thru 0x0339eb (0xa919 bytes) 000042 1787388998.272870 mapread 0x0164ce thru 0x01a17d (0x3cb0 bytes) 000044 1787388998.316533 write 0x01c172 thru 0x023f9f (0x7e2e bytes) 000046 1787388998.331913 mapread 0x0006fd thru 0x00a4b4 (0x9db8 bytes) 000048 1787388998.411788 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) 000049 1787388998.765799 mapwrite 0x01a7a7 thru 0x01bdd1 (0x162b bytes) 000050 1787388998.891466 mapread 0x00a1b7 thru 0x019d80 (0xfbca bytes) 000051 1787388998.907248 write 0x02d20f thru 0x02e0bc (0xeae bytes) 000052 1787388998.915721 mapwrite 0x02d724 thru 0x03cec7 (0xf7a4 bytes) 000058 1787388999.029264 write 0x0042c6 thru 0x010cf0 (0xca2b bytes) 000062 1787388999.033914 mapwrite 0x03d508 thru 0x03ffff (0x2af8 bytes) 000063 1787388999.097821 mapread 0x001011 thru 0x0073a7 (0x6397 bytes) 000065 1787388999.104447 write 0x01b121 thru 0x0232b3 (0x8193 bytes) 000067 1787388999.110202 trunc from 0x040000 to 0x000fab (0x3f054 bytes) 000068 1787388999.128687 MIRROR_EXTEND to 2 mirrors 000071 1787388999.795465 trunc from 0x000fac to 0x03ae1f (0x39e74 bytes) 000072 1787388999.885071 write 0x0074f8 thru 0x016333 (0xee3c bytes) 000074 1787388999.927986 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) 000075 1787389000.256753 mapwrite 0x0213c3 thru 0x022046 (0xc84 bytes) 000077 1787389000.385945 mapwrite 0x0254f4 thru 0x029cab (0x47b8 bytes) 000078 1787389000.593170 read 0x00c9d3 thru 0x00e267 (0x1895 bytes) 000080 1787389000.605741 trunc from 0x03ae20 to 0x01953e (0x218e1 bytes) 000081 1787389000.664527 trunc from 0x01953f to 0x016fa5 (0x2599 bytes) 000082 1787389000.695708 write 0x03513b thru 0x03ffff (0xaec5 bytes) 000083 1787389000.709131 read 0x000cd4 thru 0x00a811 (0x9b3e bytes) 000085 1787389000.733784 trunc from 0x040000 to 0x001908 (0x3e6f7 bytes) 000090 1787389000.795948 MIRROR_EXTEND to 2 mirrors 000091 1787389001.435346 write 0x00d931 thru 0x01d1b0 (0xf880 bytes) 000092 1787389001.504225 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1d1b1): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d1b1): Mirror 2: 0xc4218d82 (stale) CRC-32 checksum value for chunk [0, 0x1d1b1): Mirror 1: 0xb7361b30 Mirror 3: 0xb7361b30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d1b1: skipped 000093 1787389002.868052 mapwrite 0x0309d4 thru 0x034bfd (0x422a bytes) 000094 1787389003.268266 read 0x02fdb6 thru 0x0347e7 (0x4a32 bytes) 000095 1787389003.269003 mapwrite 0x011cd7 thru 0x012292 (0x5bc bytes) 000096 1787389003.363687 mapwrite 0x035452 thru 0x03bc76 (0x6825 bytes) 000099 1787389003.496528 mapwrite 0x03ad41 thru 0x03ffff (0x52bf bytes) 000100 1787389003.678422 trunc from 0x040000 to 0x0341bd (0xbe42 bytes) 000102 1787389003.724511 mapread 0x02f2ae thru 0x0341bd (0x4f10 bytes) 000103 1787389003.745406 MIRROR_RESYNC 3 mirrors 000104 1787389005.232468 MIRROR_RESYNC 3 mirrors 000105 1787389005.794306 mapread 0x027542 thru 0x0341bd (0xcc7c bytes) 000106 1787389005.830631 mapwrite 0x0029b0 thru 0x007569 (0x4bba bytes) 000108 1787389006.165019 trunc from 0x0341be to 0x00034e (0x33e6f bytes) 000109 1787389006.215639 write 0x0332ff thru 0x03fc72 (0xc974 bytes) 000110 1787389006.218402 write 0x011249 thru 0x01aad1 (0x9889 bytes) 000112 1787389006.244248 mapread 0x000168 thru 0x00f832 (0xf6cb bytes) 000113 1787389006.257814 write 0x031f3b thru 0x03aa64 (0x8b2a bytes) 000115 1787389006.282251 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fc73): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc73): Mirror 2: 0x8a6ea0ac (stale) Mirror 3: 0x8a6ea0ac (stale) CRC-32 checksum value for chunk [0, 0x3fc73): Mirror 1: 0x42fa9457 Mirror 4: 0x42fa9457 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc73: skipped 000116 1787389007.587986 mapread 0x0289b9 thru 0x02cf3f (0x4587 bytes) 000120 1787389007.651168 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc73): Mirror 3: 0x8a6ea0ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc73): Mirror 3: 0x8a6ea0ac (stale) CRC-32 checksum value for chunk [0, 0x3fc73): Mirror 1: 0x42fa9457 Mirror 4: 0x42fa9457 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc73: skipped 000122 1787389008.698109 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fc73): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc73): Mirror 3: 0x8a6ea0ac (stale) CRC-32 checksum value for chunk [0, 0x3fc73): Mirror 1: 0x42fa9457 Mirror 4: 0x42fa9457 Mirror 5: 0x42fa9457 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc73: skipped 000123 1787389010.173448 read 0x02ac15 thru 0x034b7d (0x9f69 bytes) 000124 1787389010.202627 trunc from 0x03fc73 to 0x016121 (0x29b51 bytes) 000128 1787389010.365123 read 0x00adf1 thru 0x016121 (0xb331 bytes) 000129 1787389010.390666 read 0x005c13 thru 0x006273 (0x661 bytes) 000130 1787389010.396702 write 0x03645d thru 0x03ffff (0x9ba3 bytes) 000131 1787389010.412473 trunc from 0x040000 to 0x01e6f1 (0x2190e bytes) 000133 1787389010.441816 write 0x0099a0 thru 0x015c31 (0xc292 bytes) 000137 1787389010.486492 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e6f2): Mirror 4: 0x97512159 (stale) Mirror 5: 0x97512159 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e6f2): Mirror 4: 0x97512159 (stale) Mirror 5: 0x97512159 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e6f2: skipped 000141 1787389011.272328 write 0x02ecaa thru 0x038e03 (0xa15a bytes) 000142 1787389011.278228 read 0x00a225 thru 0x010c61 (0x6a3d bytes) 000143 1787389011.299103 read 0x001dd6 thru 0x008ace (0x6cf9 bytes) 000146 1787389011.319598 write 0x031a82 thru 0x03d2e5 (0xb864 bytes) 000148 1787389011.329151 trunc from 0x03d2e6 to 0x02b5d7 (0x11d0e bytes) 000149 1787389011.387745 MIRROR_RESYNC 3 mirrors 000150 1787389012.985911 trunc from 0x02b5d8 to 0x02f2b2 (0x3cdb bytes) 000152 1787389013.154722 mapread 0x01853a thru 0x020ee8 (0x89af bytes) 000155 1787389013.210186 mapread 0x010290 thru 0x016029 (0x5d9a bytes) 000159 1787389013.254969 trunc from 0x02f2b3 to 0x033e74 (0x4bc2 bytes) 000160 1787389013.287605 read 0x009007 thru 0x00b298 (0x2292 bytes) 000161 1787389013.289072 read 0x01b8d0 thru 0x024765 (0x8e96 bytes) 000163 1787389013.313405 MIRROR_RESYNC 3 mirrors 000164 1787389014.171104 write 0x0048d6 thru 0x00e862 (0x9f8d bytes) 000167 1787389014.330586 MIRROR_RESYNC 3 mirrors 000168 1787389015.269275 mapwrite 0x024e2d thru 0x025a2f (0xc03 bytes) 000170 1787389015.513710 mapread 0x020396 thru 0x0263d3 (0x603e bytes) 000172 1787389015.525585 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33e75): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e75): Mirror 4: 0xd8467f7 (stale) Mirror 5: 0xd8467f7 (stale) CRC-32 checksum value for chunk [0, 0x33e75): Mirror 1: 0xa53efc8c Mirror 6: 0xa53efc8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e75: skipped 000176 1787389016.591425 MIRROR_RESYNC 4 mirrors 000178 1787389017.545596 read 0x01f20a thru 0x02dc22 (0xea19 bytes) 000179 1787389017.562074 write 0x03d467 thru 0x03ffff (0x2b99 bytes) 000181 1787389017.669917 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7a3ba065 (stale) Mirror 5: 0x7a3ba065 (stale) Mirror 6: 0x7a3ba065 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x56a8e765 Mirror 7: 0x56a8e765 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000182 1787389019.073569 write 0x0058e8 thru 0x00f41c (0x9b35 bytes) 000186 1787389019.184588 write 0x025baa thru 0x02f764 (0x9bbb bytes) 000187 1787389019.210829 write 0x03e933 thru 0x03ffff (0x16cd bytes) 000189 1787389019.222440 read 0x038f6a thru 0x03ffff (0x7096 bytes) 000192 1787389019.234005 trunc from 0x040000 to 0x02cd3d (0x132c2 bytes) 000193 1787389019.301912 read 0x01b299 thru 0x022ff5 (0x7d5d bytes) 000195 1787389019.347839 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cd3e): Mirror 4: 0x4205bddc (stale) Mirror 5: 0x4205bddc (stale) Mirror 6: 0x4205bddc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd3e): Mirror 4: 0x4205bddc (stale) Mirror 5: 0x4205bddc (stale) Mirror 6: 0x4205bddc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd3e: skipped 000196 1787389020.194771 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2cd3e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd3e): Mirror 4: 0x4205bddc (stale) Mirror 5: 0x4205bddc (stale) Mirror 6: 0x4205bddc (stale) CRC-32 checksum value for chunk [0, 0x2cd3e): Mirror 1: 0x4e8c0a2c Mirror 7: 0x4e8c0a2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd3e: skipped 000198 1787389021.536257 mapread 0x00bb3e thru 0x00f7f2 (0x3cb5 bytes) 000199 1787389021.605569 mapread 0x01f003 thru 0x0274a4 (0x84a2 bytes) 000201 1787389021.689069 read 0x01ec48 thru 0x028ec5 (0xa27e bytes) 000202 1787389021.721004 mapwrite 0x031fac thru 0x03b748 (0x979d bytes) 000203 1787389022.036079 write 0x011d61 thru 0x01f105 (0xd3a5 bytes) 000204 1787389022.048895 read 0x00a4cd thru 0x0113dd (0x6f11 bytes) 000205 1787389022.061559 write 0x02642b thru 0x0295d0 (0x31a6 bytes) 000206 1787389022.064569 mapread 0x024a68 thru 0x02b713 (0x6cac bytes) 000209 1787389022.067226 write 0x00168d thru 0x00b05b (0x99cf bytes) 000210 1787389022.075516 MIRROR_RESYNC 5 mirrors 000211 1787389023.561315 write 0x033845 thru 0x03a98e (0x714a bytes) 000212 1787389023.646564 trunc from 0x03b749 to 0x01c04d (0x1f6fb bytes) 000213 1787389023.680694 mapwrite 0x00cda4 thru 0x016dde (0xa03b bytes) 000214 1787389023.771593 MIRROR_RESYNC 5 mirrors 000215 1787389024.966032 mapwrite 0x034926 thru 0x03e6d6 (0x9db1 bytes) 000216 1787389025.303655 write 0x016b8f thru 0x025a5e (0xeed0 bytes) 000218 1787389025.330444 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e6d7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6d7): Mirror 4: 0x27873548 (stale) Mirror 5: 0x27873548 (stale) Mirror 6: 0x27873548 (stale) Mirror 7: 0x27873548 (stale) CRC-32 checksum value for chunk [0, 0x3e6d7): Mirror 1: 0x54909672 Mirror 8: 0x54909672 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6d7: skipped 000219 1787389026.503032 write 0x0145ec thru 0x01d6fe (0x9113 bytes) 000220 1787389026.602217 mapread 0x018e54 thru 0x01b122 (0x22cf bytes) 000221 1787389026.603356 read 0x00af6e thru 0x01a73a (0xf7cd bytes) 000225 1787389026.614126 write 0x024b1b thru 0x02f8f9 (0xaddf bytes) 000226 1787389026.628430 write 0x0285bc thru 0x038058 (0xfa9d bytes) 000227 1787389026.635666 trunc from 0x03e6d7 to 0x0301f1 (0xe4e5 bytes) 000229 1787389026.686172 mapread 0x00a2bd thru 0x00e8ed (0x4631 bytes) 000230 1787389026.691320 mapread 0x01c7fa thru 0x0245cb (0x7dd2 bytes) 000231 1787389026.711738 read 0x0052be thru 0x008ac2 (0x3805 bytes) 000233 1787389026.722599 mapread 0x01d736 thru 0x029ce9 (0xc5b4 bytes) 000234 1787389026.724663 trunc from 0x0301f2 to 0x0309a2 (0x7b1 bytes) 000235 1787389026.752127 mapread 0x03090f thru 0x0309a2 (0x94 bytes) 000240 1787389026.765329 read 0x00c147 thru 0x014781 (0x863b bytes) 000241 1787389026.766280 trunc from 0x0309a3 to 0x03426b (0x38c9 bytes) 000242 1787389026.786839 mapread 0x0093aa thru 0x018a9b (0xf6f2 bytes) 000244 1787389026.791789 read 0x008880 thru 0x00914f (0x8d0 bytes) 000245 1787389026.793541 read 0x00118f thru 0x005562 (0x43d4 bytes) 000246 1787389026.794357 mapread 0x020a8a thru 0x02e5c9 (0xdb40 bytes) 000251 1787389026.801629 mapwrite 0x008afc thru 0x009950 (0xe55 bytes) 000253 1787389026.894658 mapwrite 0x00521f thru 0x00e7ce (0x95b0 bytes) 000255 1787389027.031889 write 0x00d68e thru 0x016972 (0x92e5 bytes) 000256 1787389027.061469 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3426c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3426c): Mirror 4: 0x396b7f65 (stale) Mirror 5: 0x396b7f65 (stale) Mirror 6: 0x396b7f65 (stale) Mirror 7: 0x396b7f65 (stale) Mirror 8: 0xeeb1b217 (stale) CRC-32 checksum value for chunk [0, 0x3426c): Mirror 1: 0x9fb86123 Mirror 9: 0x9fb86123 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3426c: skipped 000257 1787389028.096777 write 0x00945e thru 0x0149f6 (0xb599 bytes) 000259 1787389028.224288 mapread 0x01b8ff thru 0x029670 (0xdd72 bytes) 000262 1787389028.266192 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3426c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3426c): Mirror 4: 0x396b7f65 (stale) Mirror 5: 0x396b7f65 (stale) Mirror 6: 0x396b7f65 (stale) Mirror 7: 0x396b7f65 (stale) Mirror 8: 0xeeb1b217 (stale) Mirror 9: 0x9fb86123 (stale) CRC-32 checksum value for chunk [0, 0x3426c): Mirror 1: 0x9967f8d8 Mirror 10: 0x9967f8d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3426c: skipped 000263 1787389029.175184 mapread 0x02f0d6 thru 0x02f9da (0x905 bytes) 000267 1787389029.211813 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3426c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3426c): Mirror 4: 0x396b7f65 (stale) Mirror 5: 0x396b7f65 (stale) Mirror 6: 0x396b7f65 (stale) Mirror 7: 0x396b7f65 (stale) Mirror 8: 0xeeb1b217 (stale) Mirror 9: 0x9fb86123 (stale) CRC-32 checksum value for chunk [0, 0x3426c): Mirror 1: 0x9967f8d8 Mirror 10: 0x9967f8d8 Mirror 11: 0x9967f8d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3426c: skipped 000268 1787389030.704692 write 0x02d668 thru 0x02f07e (0x1a17 bytes) 000269 1787389030.861729 trunc from 0x03426c to 0x033725 (0xb46 bytes) 000270 1787389030.920811 trunc from 0x033726 to 0x030b04 (0x2c21 bytes) 000271 1787389030.946915 MIRROR_RESYNC 9 mirrors 000274 1787389032.798406 write 0x005a3e thru 0x01238c (0xc94f bytes) 000275 1787389032.896734 read 0x023769 thru 0x02aa28 (0x72c0 bytes) 000276 1787389032.912201 mapread 0x006caf thru 0x00e4ad (0x77ff bytes) 000277 1787389032.914308 mapwrite 0x009a77 thru 0x0179d9 (0xdf63 bytes) 000278 1787389033.051518 trunc from 0x030b05 to 0x03b03b (0xa537 bytes) 000279 1787389033.098098 trunc from 0x03b03c to 0x03443e (0x6bfd bytes) 000280 1787389033.147302 trunc from 0x03443f to 0x0305c1 (0x3e7d bytes) 000282 1787389033.191929 trunc from 0x0305c2 to 0x015acd (0x1aaf4 bytes) 000285 1787389033.237730 trunc from 0x015ace to 0x011a4e (0x407f bytes) 000288 1787389033.274072 write 0x03e702 thru 0x03ea68 (0x367 bytes) 000290 1787389033.278586 read 0x014541 thru 0x02424d (0xfd0d bytes) 000291 1787389033.296142 write 0x02a2a4 thru 0x03035f (0x60bc bytes) 000293 1787389033.303177 mapread 0x034899 thru 0x03af9c (0x6704 bytes) 000294 1787389033.308620 read 0x034204 thru 0x03aac3 (0x68c0 bytes) 000295 1787389033.311754 mapread 0x0002f5 thru 0x00084e (0x55a bytes) 000296 1787389033.314521 trunc from 0x03ea69 to 0x011fde (0x2ca8a bytes) 000297 1787389033.337452 write 0x02ccef thru 0x031846 (0x4b58 bytes) 000298 1787389033.340226 mapread 0x01ebe7 thru 0x01fe0e (0x1228 bytes) 000299 1787389033.362238 mapread 0x01e6a0 thru 0x029666 (0xafc7 bytes) 000303 1787389033.384997 write 0x003e5b thru 0x00817b (0x4321 bytes) 000304 1787389033.386297 write 0x010d20 thru 0x01bd89 (0xb06a bytes) 000305 1787389033.390151 write 0x034029 thru 0x03ffff (0xbfd7 bytes) 000306 1787389033.393058 read 0x01b046 thru 0x01dd38 (0x2cf3 bytes) 000307 1787389033.394246 read 0x01a1f6 thru 0x02247e (0x8289 bytes) 000309 1787389033.397353 mapwrite 0x02ee81 thru 0x031998 (0x2b18 bytes) 000310 1787389033.547212 MIRROR_RESYNC 9 mirrors 000311 1787389035.464631 mapread 0x02ccbd thru 0x03262d (0x5971 bytes) 000312 1787389035.486115 mapread 0x01c3fc thru 0x027104 (0xad09 bytes) 000318 1787389035.503667 trunc from 0x040000 to 0x004cde (0x3b321 bytes) 000320 1787389035.636304 write 0x0136ba thru 0x02331e (0xfc65 bytes) 000321 1787389035.639251 mapwrite 0x01ddfc thru 0x01ea47 (0xc4c bytes) 000322 1787389035.773121 MIRROR_RESYNC 9 mirrors 000323 1787389038.447247 mapwrite 0x014e8c thru 0x01576e (0x8e3 bytes) 000325 1787389038.751391 mapread 0x013c05 thru 0x020c12 (0xd00e bytes) 000326 1787389038.773268 read 0x0008fb thru 0x0046d9 (0x3ddf bytes) 000327 1787389038.788407 trunc from 0x02331f to 0x011810 (0x11b0e bytes) 000328 1787389038.838007 MIRROR_RESYNC 9 mirrors 000330 1787389040.303256 write 0x036cc5 thru 0x03ffff (0x933b bytes) 000332 1787389040.406153 mapwrite 0x02e6ab thru 0x035c27 (0x757d bytes) 000333 1787389040.570590 mapwrite 0x0117d9 thru 0x020e96 (0xf6be bytes) 000334 1787389040.733096 MIRROR_RESYNC 9 mirrors 000336 1787389042.509111 trunc from 0x040000 to 0x018ed1 (0x2712e bytes) 000337 1787389042.606436 mapread 0x003150 thru 0x0051b8 (0x2069 bytes) 000338 1787389042.624646 read 0x005fdb thru 0x00934c (0x3372 bytes) 000339 1787389042.625506 mapread 0x0159ef thru 0x015f9b (0x5ad bytes) 000341 1787389042.645375 mapwrite 0x0201a4 thru 0x0201a9 (0x6 bytes) 000342 1787389042.739418 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x201aa): Mirror 4: 0xfb2bc1ef (stale) Mirror 5: 0xfb2bc1ef (stale) Mirror 7: 0xfb2bc1ef (stale) Mirror 8: 0xfb2bc1ef (stale) Mirror 9: 0xfb2bc1ef (stale) Mirror 10: 0xfb2bc1ef (stale) Mirror 11: 0xfb2bc1ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x201aa): Mirror 4: 0xfb2bc1ef (stale) Mirror 5: 0xfb2bc1ef (stale) Mirror 7: 0xfb2bc1ef (stale) Mirror 8: 0xfb2bc1ef (stale) Mirror 9: 0xfb2bc1ef (stale) Mirror 10: 0xfb2bc1ef (stale) Mirror 11: 0xfb2bc1ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x201aa: skipped 000344 1787389043.425097 MIRROR_RESYNC 8 mirrors 000345 1787389045.329318 trunc from 0x0201aa to 0x035169 (0x14fc0 bytes) 000346 1787389045.401127 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3516a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3516a): Mirror 4: 0xd3dbd98b (stale) Mirror 5: 0xd3dbd98b (stale) Mirror 7: 0xd3dbd98b (stale) Mirror 8: 0xd3dbd98b (stale) Mirror 9: 0xd3dbd98b (stale) Mirror 10: 0xd3dbd98b (stale) Mirror 11: 0xd3dbd98b (stale) CRC-32 checksum value for chunk [0, 0x3516a): Mirror 1: 0xd3dbd98b Mirror 12: 0xd3dbd98b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3516a: skipped 000347 1787389046.865069 trunc from 0x03516a to 0x03435f (0xe0a bytes) 000348 1787389047.009336 trunc from 0x034360 to 0x015ca7 (0x1e6b8 bytes) 000349 1787389047.032306 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x15ca8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15ca8): Mirror 4: 0xfc35465f (stale) Mirror 5: 0xfc35465f (stale) Mirror 7: 0xfc35465f (stale) Mirror 8: 0xfc35465f (stale) Mirror 9: 0xfc35465f (stale) Mirror 10: 0xfc35465f (stale) Mirror 11: 0xfc35465f (stale) Mirror 12: 0xfc35465f (stale) CRC-32 checksum value for chunk [0, 0x15ca8): Mirror 1: 0xfc35465f Mirror 13: 0xfc35465f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15ca8: skipped 000350 1787389048.376313 MIRROR_RESYNC 10 mirrors 000352 1787389049.941208 mapread 0x003577 thru 0x012664 (0xf0ee bytes) 000353 1787389049.975502 mapwrite 0x03c256 thru 0x03ffff (0x3daa bytes) 000356 1787389050.122815 read 0x03acfd thru 0x03ffff (0x5303 bytes) 000357 1787389050.123772 read 0x026b25 thru 0x030ad8 (0x9fb4 bytes) 000358 1787389050.146294 write 0x009187 thru 0x00ddf6 (0x4c70 bytes) 000360 1787389050.154009 trunc from 0x040000 to 0x035b4b (0xa4b4 bytes) 000361 1787389050.172876 mapwrite 0x005891 thru 0x01467f (0xedef bytes) 000362 1787389050.229230 write 0x03608f thru 0x03e631 (0x85a3 bytes) 000364 1787389050.232524 trunc from 0x03e632 to 0x01d320 (0x21311 bytes) 000365 1787389050.247433 trunc from 0x01d321 to 0x00c701 (0x10c1f bytes) 000366 1787389050.269725 trunc from 0x00c702 to 0x00f18b (0x2a8a bytes) 000370 1787389050.289414 mapread 0x006f07 thru 0x00ec20 (0x7d1a bytes) 000373 1787389050.302504 write 0x01220d thru 0x015de4 (0x3bd8 bytes) truncating to largest ever: 0x3f390 000374 1787389050.306380 trunc from 0x015de5 to 0x03f38f (0x295ab bytes) 000375 1787389050.326451 write 0x0133f5 thru 0x01c123 (0x8d2f bytes) 000376 1787389050.337469 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3f390): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f390): Mirror 4: 0x4b03f99b (stale) Mirror 5: 0x4b03f99b (stale) Mirror 7: 0x4b03f99b (stale) Mirror 8: 0x4b03f99b (stale) Mirror 9: 0x4b03f99b (stale) Mirror 10: 0x4b03f99b (stale) Mirror 11: 0x4b03f99b (stale) Mirror 12: 0x4b03f99b (stale) Mirror 13: 0x4b03f99b (stale) CRC-32 checksum value for chunk [0, 0x3f390): Mirror 1: 0xa36cd795 Mirror 14: 0xa36cd795 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f390: skipped 000377 1787389051.569384 write 0x012564 thru 0x019362 (0x6dff bytes) 000379 1787389051.832224 trunc from 0x03f390 to 0x01b3b3 (0x23fdc bytes) 000382 1787389051.902900 mapwrite 0x01565a thru 0x0230a4 (0xda4b bytes) 000383 1787389052.082812 trunc from 0x0230a5 to 0x014f9b (0xe109 bytes) 000384 1787389052.124465 trunc from 0x014f9c to 0x022173 (0xd1d8 bytes) 000386 1787389052.165262 write 0x02eb7d thru 0x03b156 (0xc5da bytes) 000387 1787389052.175992 trunc from 0x03b157 to 0x01585b (0x258fb bytes) 000388 1787389052.220542 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1585c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1585c): Mirror 4: 0x6b5870e (stale) Mirror 5: 0x6b5870e (stale) Mirror 7: 0x6b5870e (stale) Mirror 8: 0x6b5870e (stale) Mirror 9: 0x6b5870e (stale) Mirror 10: 0x6b5870e (stale) Mirror 11: 0x6b5870e (stale) Mirror 12: 0x6b5870e (stale) Mirror 13: 0x6b5870e (stale) Mirror 14: 0x4ac18557 (stale) CRC-32 checksum value for chunk [0, 0x1585c): Mirror 1: 0x1c0da873 Mirror 15: 0x1c0da873 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1585c: skipped 000390 1787389053.595796 mapread 0x014e66 thru 0x01585b (0x9f6 bytes) 000391 1787389053.696917 write 0x019b4e thru 0x023b03 (0x9fb6 bytes) 000392 1787389053.877416 write 0x006bb6 thru 0x00f57e (0x89c9 bytes) 000394 1787389053.895139 mapwrite 0x006c5a thru 0x009f61 (0x3308 bytes) 000399 1787389053.967310 write 0x0358ef thru 0x03ffff (0xa711 bytes) 000400 1787389053.976031 read 0x02974c thru 0x02e031 (0x48e6 bytes) 000401 1787389053.994517 write 0x011ce8 thru 0x01af5f (0x9278 bytes) 000403 1787389054.015120 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd698497f (stale) Mirror 5: 0xd698497f (stale) Mirror 7: 0xd698497f (stale) Mirror 8: 0xd698497f (stale) Mirror 9: 0xd698497f (stale) Mirror 10: 0xd698497f (stale) Mirror 11: 0xd698497f (stale) Mirror 12: 0xd698497f (stale) Mirror 13: 0xd698497f (stale) Mirror 14: 0x7741822f (stale) Mirror 15: 0x3cf48762 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd3f9632c Mirror 16: 0xd3f9632c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000404 1787389055.727988 read 0x022c44 thru 0x022d53 (0x110 bytes) 000407 1787389055.765325 write 0x02fec4 thru 0x03fd43 (0xfe80 bytes) 000408 1787389055.937369 read 0x0195cb thru 0x01a087 (0xabd bytes) 000410 1787389055.978093 MIRROR_RESYNC 13 mirrors 000411 1787389058.479313 write 0x01ef1c thru 0x01fae7 (0xbcc bytes) 000412 1787389058.713104 MIRROR_RESYNC 13 mirrors 000414 1787389061.360020 read 0x019987 thru 0x0242c4 (0xa93e bytes) 000415 1787389061.390660 MIRROR_RESYNC 13 mirrors 000416 1787389062.408212 mapread 0x020d2c thru 0x02ebab (0xde80 bytes) 000418 1787389062.433030 trunc from 0x040000 to 0x0396e9 (0x6916 bytes) 000420 1787389062.561989 mapwrite 0x01d400 thru 0x0282ea (0xaeeb bytes) 000421 1787389062.663722 mapwrite 0x02281f thru 0x02832a (0x5b0c bytes) 000422 1787389062.715301 mapwrite 0x0146f8 thru 0x01f6ca (0xafd3 bytes) 000423 1787389062.774305 trunc from 0x0396ea to 0x0380d8 (0x1611 bytes) 000424 1787389062.815200 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x380d9): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380d9): Mirror 4: 0xc25488c7 (stale) Mirror 5: 0xc25488c7 (stale) Mirror 7: 0xc25488c7 (stale) Mirror 8: 0xc25488c7 (stale) Mirror 9: 0xc25488c7 (stale) Mirror 10: 0xc25488c7 (stale) Mirror 11: 0xc25488c7 (stale) Mirror 12: 0xc25488c7 (stale) Mirror 13: 0xc25488c7 (stale) Mirror 14: 0xc25488c7 (stale) Mirror 15: 0xc25488c7 (stale) Mirror 16: 0xc25488c7 (stale) CRC-32 checksum value for chunk [0, 0x380d9): Mirror 1: 0xbec4e280 Mirror 17: 0xbec4e280 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380d9: skipped 000425 1787389064.171539 trunc from 0x0380d9 to 0x0131b9 (0x24f1f bytes) 000426 1787389064.380365 mapread 0x001d3f thru 0x008bf1 (0x6eb3 bytes) 000427 1787389064.420937 write 0x014b93 thru 0x016b02 (0x1f70 bytes) 000431 1787389064.426200 write 0x00b395 thru 0x013271 (0x7edd bytes) 000434 1787389064.441161 read 0x0064ee thru 0x00cf0f (0x6a22 bytes) 000435 1787389064.442718 trunc from 0x016b03 to 0x033eb7 (0x1d3b5 bytes) 000436 1787389064.482789 mapread 0x0295fe thru 0x033eb7 (0xa8ba bytes) 000439 1787389064.520455 MIRROR_RESYNC 14 mirrors 000440 1787389067.320208 mapread 0x0189a5 thru 0x0255b9 (0xcc15 bytes) 000443 1787389067.402024 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) 000444 1787389068.243076 write 0x033173 thru 0x034e05 (0x1c93 bytes) 000445 1787389068.420401 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34e06): Mirror 4: 0xc4686ae2 (stale) Mirror 5: 0xc4686ae2 (stale) Mirror 7: 0xc4686ae2 (stale) Mirror 9: 0xc4686ae2 (stale) Mirror 10: 0xc4686ae2 (stale) Mirror 11: 0xc4686ae2 (stale) Mirror 12: 0xc4686ae2 (stale) Mirror 13: 0xc4686ae2 (stale) Mirror 14: 0xc4686ae2 (stale) Mirror 15: 0xc4686ae2 (stale) Mirror 17: 0xc4686ae2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e06): Mirror 4: 0xc4686ae2 (stale) Mirror 5: 0xc4686ae2 (stale) Mirror 7: 0xc4686ae2 (stale) Mirror 9: 0xc4686ae2 (stale) Mirror 10: 0xc4686ae2 (stale) Mirror 11: 0xc4686ae2 (stale) Mirror 12: 0xc4686ae2 (stale) Mirror 13: 0xc4686ae2 (stale) Mirror 14: 0xc4686ae2 (stale) Mirror 15: 0xc4686ae2 (stale) Mirror 17: 0xc4686ae2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e06: skipped 000446 1787389069.466489 read 0x019667 thru 0x019e3e (0x7d8 bytes) 000447 1787389069.474093 read 0x02182d thru 0x026565 (0x4d39 bytes) 000448 1787389069.493981 mapwrite 0x038a9b thru 0x03ffff (0x7565 bytes) 000449 1787389069.603409 read 0x0150f9 thru 0x01fbdb (0xaae3 bytes) 000451 1787389069.611611 mapwrite 0x03b3da thru 0x03ffff (0x4c26 bytes) 000453 1787389069.684775 read 0x003a0b thru 0x00e868 (0xae5e bytes) 000455 1787389069.687164 mapread 0x00f4f2 thru 0x010152 (0xc61 bytes) 000456 1787389069.688762 write 0x01d655 thru 0x01fa05 (0x23b1 bytes) 000457 1787389069.690629 read 0x02c9cd thru 0x032388 (0x59bc bytes) 000458 1787389069.691569 read 0x0291fa thru 0x034ba1 (0xb9a8 bytes) 000459 1787389069.692323 write 0x019b7c thru 0x025610 (0xba95 bytes) 000460 1787389069.694019 trunc from 0x040000 to 0x00587b (0x3a784 bytes) 000461 1787389069.712578 write 0x0239e8 thru 0x024a95 (0x10ae bytes) 000462 1787389069.713893 read 0x016701 thru 0x024271 (0xdb71 bytes) 000466 1787389069.725721 mapread 0x000863 thru 0x0088a6 (0x8044 bytes) 000468 1787389069.732487 MIRROR_RESYNC 12 mirrors 000469 1787389071.889832 MIRROR_EXTEND to 13 mirrors 000470 1787389072.880953 mapwrite 0x01e389 thru 0x02d9bf (0xf637 bytes) 000472 1787389073.141318 MIRROR_RESYNC 13 mirrors 000477 1787389075.189972 write 0x00e4cc thru 0x013672 (0x51a7 bytes) 000483 1787389075.337172 read 0x015729 thru 0x01b7b6 (0x608e bytes) 000486 1787389075.352629 write 0x024008 thru 0x032649 (0xe642 bytes) 000487 1787389075.362566 trunc from 0x03264a to 0x02c900 (0x5d49 bytes) 000490 1787389075.405554 read 0x02b517 thru 0x02c900 (0x13ea bytes) 000491 1787389075.410818 mapwrite 0x019ea1 thru 0x01d518 (0x3678 bytes) 000492 1787389075.519700 mapwrite 0x01b13e thru 0x02a39e (0xf261 bytes) 000494 1787389075.651110 mapread 0x0105e8 thru 0x011891 (0x12aa bytes) 000495 1787389075.652109 read 0x01d76a thru 0x027f95 (0xa82c bytes) 000496 1787389075.653480 mapwrite 0x010b53 thru 0x014048 (0x34f6 bytes) 000497 1787389075.727371 MIRROR_RESYNC 13 mirrors 000498 1787389077.669181 read 0x010ae7 thru 0x01bd00 (0xb21a bytes) 000499 1787389077.684005 read 0x01fbc7 thru 0x022283 (0x26bd bytes) 000500 1787389077.693942 trunc from 0x02c901 to 0x01248d (0x1a473 bytes) 000503 1787389077.858720 mapwrite 0x01601a thru 0x018d6f (0x2d56 bytes) 000504 1787389078.010864 mapwrite 0x006774 thru 0x00cd8e (0x661b bytes) 000505 1787389078.207627 read 0x012935 thru 0x018d6f (0x643b bytes) 000506 1787389078.232536 read 0x00d702 thru 0x018d6f (0xb66e bytes) 000507 1787389078.234381 trunc from 0x018d70 to 0x004715 (0x1465a bytes) 000508 1787389078.259729 read 0x00133f thru 0x004715 (0x33d7 bytes) 000511 1787389078.264255 trunc from 0x004716 to 0x03edd2 (0x3a6bd bytes) 000512 1787389078.296563 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3edd3): Mirror 4: 0xac15eae0 (stale) Mirror 5: 0xac15eae0 (stale) Mirror 7: 0xac15eae0 (stale) Mirror 10: 0xac15eae0 (stale) Mirror 11: 0xac15eae0 (stale) Mirror 12: 0xac15eae0 (stale) Mirror 13: 0xac15eae0 (stale) Mirror 14: 0xac15eae0 (stale) Mirror 15: 0xac15eae0 (stale) Mirror 17: 0xac15eae0 (stale) Mirror 18: 0xac15eae0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3edd3): Mirror 4: 0xac15eae0 (stale) Mirror 5: 0xac15eae0 (stale) Mirror 7: 0xac15eae0 (stale) Mirror 10: 0xac15eae0 (stale) Mirror 11: 0xac15eae0 (stale) Mirror 12: 0xac15eae0 (stale) Mirror 13: 0xac15eae0 (stale) Mirror 14: 0xac15eae0 (stale) Mirror 15: 0xac15eae0 (stale) Mirror 17: 0xac15eae0 (stale) Mirror 18: 0xac15eae0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3edd3: skipped 000513 1787389079.316228 write 0x01539a thru 0x017b7c (0x27e3 bytes) 000514 1787389079.340971 mapwrite 0x0179aa thru 0x01d619 (0x5c70 bytes) 000515 1787389079.534762 mapwrite 0x02f51e thru 0x0323aa (0x2e8d bytes) 000516 1787389079.668303 write 0x0024b4 thru 0x005ae8 (0x3635 bytes) 000517 1787389079.706145 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3edd3): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3edd3): Mirror 4: 0xac15eae0 (stale) Mirror 5: 0xac15eae0 (stale) Mirror 7: 0xac15eae0 (stale) Mirror 10: 0xac15eae0 (stale) Mirror 11: 0xac15eae0 (stale) Mirror 12: 0xac15eae0 (stale) Mirror 13: 0xac15eae0 (stale) Mirror 14: 0xac15eae0 (stale) Mirror 15: 0xac15eae0 (stale) Mirror 17: 0xac15eae0 (stale) Mirror 18: 0xac15eae0 (stale) CRC-32 checksum value for chunk [0, 0x3edd3): Mirror 1: 0x29255060 Mirror 19: 0x29255060 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3edd3: skipped 000518 1787389081.404497 read 0x011191 thru 0x015770 (0x45e0 bytes) 000519 1787389081.422685 read 0x03d93d thru 0x03edd2 (0x1496 bytes) 000520 1787389081.447379 read 0x02a785 thru 0x030906 (0x6182 bytes) 000521 1787389081.457933 mapread 0x019161 thru 0x02826f (0xf10f bytes) 000522 1787389081.479571 mapread 0x03d8bc thru 0x03edd2 (0x1517 bytes) 000523 1787389081.505686 mapwrite 0x00b4dd thru 0x018669 (0xd18d bytes) 000524 1787389081.757664 read 0x03a775 thru 0x03d25d (0x2ae9 bytes) 000525 1787389081.758416 mapwrite 0x012def thru 0x0149da (0x1bec bytes) 000527 1787389081.871900 read 0x02ed73 thru 0x03e443 (0xf6d1 bytes) 000530 1787389081.877024 trunc from 0x03edd3 to 0x01c3a1 (0x22a31 bytes) 000537 1787389081.949301 trunc from 0x01c3a2 to 0x014977 (0x7a2a bytes) 000540 1787389081.971921 mapread 0x01021e thru 0x014977 (0x475a bytes) 000542 1787389081.975990 write 0x03befc thru 0x03ffff (0x4104 bytes) 000543 1787389081.992554 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd250e8a6 (stale) Mirror 5: 0xd250e8a6 (stale) Mirror 7: 0xd250e8a6 (stale) Mirror 10: 0xd250e8a6 (stale) Mirror 11: 0xd250e8a6 (stale) Mirror 12: 0xd250e8a6 (stale) Mirror 13: 0xd250e8a6 (stale) Mirror 14: 0xd250e8a6 (stale) Mirror 15: 0xd250e8a6 (stale) Mirror 17: 0xd250e8a6 (stale) Mirror 18: 0xd250e8a6 (stale) Mirror 19: 0x185c0ec8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbafc6a57 Mirror 20: 0xbafc6a57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000544 1787389083.145336 mapwrite 0x008cc3 thru 0x012b0a (0x9e48 bytes) 000547 1787389083.568158 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd250e8a6 (stale) Mirror 5: 0xd250e8a6 (stale) Mirror 7: 0xd250e8a6 (stale) Mirror 10: 0xd250e8a6 (stale) Mirror 11: 0xd250e8a6 (stale) Mirror 12: 0xd250e8a6 (stale) Mirror 14: 0xd250e8a6 (stale) Mirror 15: 0xd250e8a6 (stale) Mirror 17: 0xd250e8a6 (stale) Mirror 18: 0xd250e8a6 (stale) Mirror 19: 0x185c0ec8 (stale) Mirror 20: 0xbafc6a57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xd250e8a6 (stale) Mirror 5: 0xd250e8a6 (stale) Mirror 7: 0xd250e8a6 (stale) Mirror 10: 0xd250e8a6 (stale) Mirror 11: 0xd250e8a6 (stale) Mirror 12: 0xd250e8a6 (stale) Mirror 14: 0xd250e8a6 (stale) Mirror 15: 0xd250e8a6 (stale) Mirror 17: 0xd250e8a6 (stale) Mirror 18: 0xd250e8a6 (stale) Mirror 19: 0x185c0ec8 (stale) Mirror 20: 0xbafc6a57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000549 1787389084.394548 read 0x01873c thru 0x01b2c5 (0x2b8a bytes) 000551 1787389084.403309 write 0x02c9f3 thru 0x0365de (0x9bec bytes) 000553 1787389084.419348 read 0x0157f0 thru 0x015a96 (0x2a7 bytes) 000558 1787389084.443526 write 0x02639a thru 0x0267bf (0x426 bytes) 000559 1787389084.453695 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd250e8a6 (stale) Mirror 7: 0xd250e8a6 (stale) Mirror 10: 0xd250e8a6 (stale) Mirror 11: 0xd250e8a6 (stale) Mirror 12: 0xd250e8a6 (stale) Mirror 14: 0xd250e8a6 (stale) Mirror 15: 0xd250e8a6 (stale) Mirror 17: 0xd250e8a6 (stale) Mirror 18: 0xd250e8a6 (stale) Mirror 19: 0x185c0ec8 (stale) Mirror 20: 0xbafc6a57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd250e8a6 (stale) Mirror 7: 0xd250e8a6 (stale) Mirror 10: 0xd250e8a6 (stale) Mirror 11: 0xd250e8a6 (stale) Mirror 12: 0xd250e8a6 (stale) Mirror 14: 0xd250e8a6 (stale) Mirror 15: 0xd250e8a6 (stale) Mirror 17: 0xd250e8a6 (stale) Mirror 18: 0xd250e8a6 (stale) Mirror 19: 0x185c0ec8 (stale) Mirror 20: 0xbafc6a57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000563 1787389085.187153 read 0x01701f thru 0x0183b9 (0x139b bytes) 000564 1787389085.194082 write 0x0110ed thru 0x013e12 (0x2d26 bytes) 000567 1787389085.213187 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd250e8a6 (stale) Mirror 7: 0xd250e8a6 (stale) Mirror 10: 0xd250e8a6 (stale) Mirror 11: 0xd250e8a6 (stale) Mirror 12: 0xd250e8a6 (stale) Mirror 15: 0xd250e8a6 (stale) Mirror 17: 0xd250e8a6 (stale) Mirror 18: 0xd250e8a6 (stale) Mirror 19: 0x185c0ec8 (stale) Mirror 20: 0xbafc6a57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd250e8a6 (stale) Mirror 7: 0xd250e8a6 (stale) Mirror 10: 0xd250e8a6 (stale) Mirror 11: 0xd250e8a6 (stale) Mirror 12: 0xd250e8a6 (stale) Mirror 15: 0xd250e8a6 (stale) Mirror 17: 0xd250e8a6 (stale) Mirror 18: 0xd250e8a6 (stale) Mirror 19: 0x185c0ec8 (stale) Mirror 20: 0xbafc6a57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000570 1787389086.106261 trunc from 0x040000 to 0x0352c2 (0xad3d bytes) 000573 1787389086.140721 mapwrite 0x01e898 thru 0x021318 (0x2a81 bytes) 000574 1787389086.298143 mapread 0x00f8d8 thru 0x01b1d5 (0xb8fe bytes) 000578 1787389086.361549 mapwrite 0x005754 thru 0x0130d7 (0xd984 bytes) 000580 1787389086.457990 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x352c3): Mirror 5: 0x26efaf6f (stale) Mirror 10: 0x26efaf6f (stale) Mirror 11: 0x26efaf6f (stale) Mirror 12: 0x26efaf6f (stale) Mirror 15: 0x26efaf6f (stale) Mirror 17: 0x26efaf6f (stale) Mirror 18: 0x26efaf6f (stale) Mirror 19: 0x4407ab7c (stale) Mirror 20: 0xe674b7d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x352c3): Mirror 5: 0x26efaf6f (stale) Mirror 10: 0x26efaf6f (stale) Mirror 11: 0x26efaf6f (stale) Mirror 12: 0x26efaf6f (stale) Mirror 15: 0x26efaf6f (stale) Mirror 17: 0x26efaf6f (stale) Mirror 18: 0x26efaf6f (stale) Mirror 19: 0x4407ab7c (stale) Mirror 20: 0xe674b7d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x352c3: skipped 000581 1787389087.426520 trunc from 0x0352c3 to 0x0293cb (0xbef7 bytes) 000582 1787389087.450801 trunc from 0x0293cc to 0x027dd3 (0x15f8 bytes) 000585 1787389087.485380 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27dd4): Mirror 5: 0xdf3f0418 (stale) Mirror 10: 0xdf3f0418 (stale) Mirror 11: 0xdf3f0418 (stale) Mirror 12: 0xdf3f0418 (stale) Mirror 15: 0xdf3f0418 (stale) Mirror 17: 0xdf3f0418 (stale) Mirror 19: 0xde028d43 (stale) Mirror 20: 0xf224adff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27dd4): Mirror 5: 0xdf3f0418 (stale) Mirror 10: 0xdf3f0418 (stale) Mirror 11: 0xdf3f0418 (stale) Mirror 12: 0xdf3f0418 (stale) Mirror 15: 0xdf3f0418 (stale) Mirror 17: 0xdf3f0418 (stale) Mirror 19: 0xde028d43 (stale) Mirror 20: 0xf224adff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27dd4: skipped 000588 1787389088.361178 read 0x01562c thru 0x01dc5b (0x8630 bytes) 000589 1787389088.371464 mapwrite 0x010649 thru 0x014f1c (0x48d4 bytes) 000590 1787389088.469576 mapwrite 0x020da7 thru 0x02fe37 (0xf091 bytes) 000591 1787389088.630547 mapread 0x0043b9 thru 0x0094fc (0x5144 bytes) 000593 1787389088.641665 read 0x026bd1 thru 0x02fe37 (0x9267 bytes) 000594 1787389088.642696 read 0x017cfe thru 0x024866 (0xcb69 bytes) 000597 1787389088.645265 trunc from 0x02fe38 to 0x021188 (0xecaf bytes) 000598 1787389088.662222 mapread 0x003d70 thru 0x008c87 (0x4f18 bytes) 000601 1787389088.666244 trunc from 0x021189 to 0x02c27f (0xb0f7 bytes) 000603 1787389088.694097 mapread 0x01e72f thru 0x027430 (0x8d02 bytes) 000606 1787389088.712693 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2c280): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c280): Mirror 5: 0x7a0a16cf (stale) Mirror 10: 0x7a0a16cf (stale) Mirror 11: 0x7a0a16cf (stale) Mirror 12: 0x7a0a16cf (stale) Mirror 15: 0x7a0a16cf (stale) Mirror 17: 0x7a0a16cf (stale) Mirror 19: 0xccf1251e (stale) Mirror 20: 0x4f87efca (stale) CRC-32 checksum value for chunk [0, 0x2c280): Mirror 1: 0xdcf0d412 Mirror 21: 0xdcf0d412 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c280: skipped 000608 1787389090.017522 write 0x02b2cb thru 0x03a1cf (0xef05 bytes) 000609 1787389090.242481 read 0x01409d thru 0x015c49 (0x1bad bytes) 000611 1787389090.255210 read 0x014305 thru 0x020705 (0xc401 bytes) 000612 1787389090.268535 mapwrite 0x0123e3 thru 0x01a48c (0x80aa bytes) 000613 1787389090.352572 write 0x01da7e thru 0x0252ef (0x7872 bytes) 000616 1787389090.359618 read 0x035cb8 thru 0x03a1cf (0x4518 bytes) 000618 1787389090.362003 mapwrite 0x00ea3c thru 0x014e5f (0x6424 bytes) 000619 1787389090.426268 MIRROR_RESYNC 10 mirrors 000620 1787389092.454522 write 0x017a21 thru 0x023908 (0xbee8 bytes) 000625 1787389092.567554 read 0x02fee9 thru 0x03a1cf (0xa2e7 bytes) 000627 1787389092.607755 trunc from 0x03a1d0 to 0x029236 (0x10f99 bytes) 000629 1787389092.659323 mapwrite 0x01c767 thru 0x020c5d (0x44f7 bytes) 000631 1787389092.756513 write 0x034f0d thru 0x03ffff (0xb0f3 bytes) 000632 1787389092.761687 mapread 0x00aa6a thru 0x016f0d (0xc4a4 bytes) 000633 1787389092.806923 read 0x0333b9 thru 0x03ffff (0xcc47 bytes) 000634 1787389092.830027 read 0x0190a4 thru 0x02715e (0xe0bb bytes) 000637 1787389092.835166 write 0x02e045 thru 0x0337f2 (0x57ae bytes) 000640 1787389092.840227 read 0x00c85d thru 0x010b2e (0x42d2 bytes) 000641 1787389092.841891 read 0x0137bd thru 0x013d9b (0x5df bytes) 000642 1787389092.843056 trunc from 0x040000 to 0x01417d (0x2be82 bytes) 000643 1787389092.864447 mapread 0x004fa5 thru 0x009ed4 (0x4f30 bytes) 000644 1787389092.865824 trunc from 0x01417e to 0x01be89 (0x7d0c bytes) 000646 1787389092.900316 read 0x004835 thru 0x00f78e (0xaf5a bytes) 000648 1787389092.901982 trunc from 0x01be8a to 0x03703a (0x1b1b1 bytes) 000650 1787389092.961502 trunc from 0x03703b to 0x01aa35 (0x1c605 bytes) 000651 1787389093.011528 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1aa36): Mirror 5: 0x2e7273b2 (stale) Mirror 10: 0x2e7273b2 (stale) Mirror 11: 0x2e7273b2 (stale) Mirror 12: 0x2e7273b2 (stale) Mirror 15: 0x2e7273b2 (stale) Mirror 17: 0x2e7273b2 (stale) Mirror 19: 0x2e7273b2 (stale) Mirror 20: 0x2e7273b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1aa36): Mirror 5: 0x2e7273b2 (stale) Mirror 10: 0x2e7273b2 (stale) Mirror 11: 0x2e7273b2 (stale) Mirror 12: 0x2e7273b2 (stale) Mirror 15: 0x2e7273b2 (stale) Mirror 17: 0x2e7273b2 (stale) Mirror 19: 0x2e7273b2 (stale) Mirror 20: 0x2e7273b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1aa36: skipped 000652 1787389093.858743 write 0x005e58 thru 0x00a62a (0x47d3 bytes) 000653 1787389093.872752 mapread 0x015f18 thru 0x01aa35 (0x4b1e bytes) 000657 1787389093.903750 trunc from 0x01aa36 to 0x038383 (0x1d94e bytes) 000658 1787389093.949702 mapwrite 0x032173 thru 0x03cdb5 (0xac43 bytes) 000659 1787389094.065469 write 0x035c38 thru 0x03b015 (0x53de bytes) 000660 1787389094.068672 mapread 0x00cada thru 0x016dd8 (0xa2ff bytes) 000661 1787389094.083391 mapwrite 0x00137c thru 0x004355 (0x2fda bytes) 000664 1787389094.161616 trunc from 0x03cdb6 to 0x02cc96 (0x1011f bytes) 000666 1787389094.197175 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cc97): Mirror 5: 0x7f82b8e4 (stale) Mirror 10: 0x7f82b8e4 (stale) Mirror 11: 0x7f82b8e4 (stale) Mirror 12: 0x7f82b8e4 (stale) Mirror 15: 0x7f82b8e4 (stale) Mirror 19: 0x7f82b8e4 (stale) Mirror 20: 0x7f82b8e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cc97): Mirror 5: 0x7f82b8e4 (stale) Mirror 10: 0x7f82b8e4 (stale) Mirror 11: 0x7f82b8e4 (stale) Mirror 12: 0x7f82b8e4 (stale) Mirror 15: 0x7f82b8e4 (stale) Mirror 19: 0x7f82b8e4 (stale) Mirror 20: 0x7f82b8e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cc97: skipped 000668 1787389094.779463 read 0x017ddb thru 0x01c9dc (0x4c02 bytes) 000669 1787389094.790556 mapread 0x02be51 thru 0x02cc96 (0xe46 bytes) 000672 1787389094.809238 trunc from 0x02cc97 to 0x00af4a (0x21d4c bytes) 000673 1787389094.829535 write 0x014128 thru 0x023c49 (0xfb22 bytes) 000674 1787389094.833422 read 0x019184 thru 0x021c54 (0x8ad1 bytes) 000675 1787389094.835369 mapread 0x022b82 thru 0x023c49 (0x10c8 bytes) 000680 1787389094.843670 mapwrite 0x030811 thru 0x035ff3 (0x57e3 bytes) 000681 1787389094.929637 mapwrite 0x030ea8 thru 0x03ffff (0xf158 bytes) 000682 1787389095.023157 mapwrite 0x022451 thru 0x02d523 (0xb0d3 bytes) 000684 1787389095.148366 read 0x03963a thru 0x03a715 (0x10dc bytes) 000687 1787389095.163177 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x45a43726 (stale) Mirror 10: 0x45a43726 (stale) Mirror 11: 0x45a43726 (stale) Mirror 12: 0x45a43726 (stale) Mirror 15: 0x45a43726 (stale) Mirror 19: 0x45a43726 (stale) Mirror 20: 0x45a43726 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4c168813 Mirror 21: 0x4c168813 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000688 1787389096.260002 mapread 0x035144 thru 0x037ecd (0x2d8a bytes) 000691 1787389096.298522 trunc from 0x040000 to 0x012beb (0x2d414 bytes) 000692 1787389096.416807 trunc from 0x012bec to 0x03aea4 (0x282b9 bytes) 000693 1787389096.442233 mapwrite 0x00a4e3 thru 0x013fda (0x9af8 bytes) 000694 1787389096.547909 trunc from 0x03aea5 to 0x03c997 (0x1af3 bytes) 000697 1787389096.569486 mapwrite 0x0398a5 thru 0x03ffff (0x675b bytes) 000699 1787389096.703619 mapread 0x00ede7 thru 0x01e77d (0xf997 bytes) 000700 1787389096.720543 write 0x02e87a thru 0x034ae0 (0x6267 bytes) 000701 1787389096.752436 write 0x02d908 thru 0x03759f (0x9c98 bytes) 000702 1787389096.763976 mapread 0x01a31b thru 0x023a1c (0x9702 bytes) 000703 1787389096.766120 mapread 0x00cf57 thru 0x00dae9 (0xb93 bytes) 000704 1787389096.767317 trunc from 0x040000 to 0x0276e4 (0x1891b bytes) 000707 1787389096.815478 mapwrite 0x014873 thru 0x017bb0 (0x333e bytes) 000708 1787389096.888536 MIRROR_RESYNC 9 mirrors 000709 1787389098.456557 write 0x020473 thru 0x02f94a (0xf4d8 bytes) 000710 1787389098.540489 mapwrite 0x0112d0 thru 0x015910 (0x4641 bytes) 000711 1787389098.623714 write 0x004ded thru 0x0125c3 (0xd7d7 bytes) 000712 1787389098.630480 mapwrite 0x019fff thru 0x01c9e5 (0x29e7 bytes) 000714 1787389098.712593 mapread 0x00b550 thru 0x00ffe2 (0x4a93 bytes) 000715 1787389098.714330 mapwrite 0x014a58 thru 0x015e6a (0x1413 bytes) 000717 1787389098.846005 mapwrite 0x03fc71 thru 0x03ffff (0x38f bytes) 000718 1787389098.986578 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x955dc1b0 (stale) Mirror 10: 0x955dc1b0 (stale) Mirror 11: 0x955dc1b0 (stale) Mirror 12: 0x955dc1b0 (stale) Mirror 15: 0x955dc1b0 (stale) Mirror 20: 0x955dc1b0 (stale) Mirror 21: 0x955dc1b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x955dc1b0 (stale) Mirror 10: 0x955dc1b0 (stale) Mirror 11: 0x955dc1b0 (stale) Mirror 12: 0x955dc1b0 (stale) Mirror 15: 0x955dc1b0 (stale) Mirror 20: 0x955dc1b0 (stale) Mirror 21: 0x955dc1b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000719 1787389099.687428 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x955dc1b0 (stale) Mirror 11: 0x955dc1b0 (stale) Mirror 12: 0x955dc1b0 (stale) Mirror 15: 0x955dc1b0 (stale) Mirror 20: 0x955dc1b0 (stale) Mirror 21: 0x955dc1b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x955dc1b0 (stale) Mirror 11: 0x955dc1b0 (stale) Mirror 12: 0x955dc1b0 (stale) Mirror 15: 0x955dc1b0 (stale) Mirror 20: 0x955dc1b0 (stale) Mirror 21: 0x955dc1b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000720 1787389100.359550 trunc from 0x040000 to 0x015af7 (0x2a508 bytes) 000721 1787389100.382441 mapwrite 0x02f20d thru 0x03d6ae (0xe4a2 bytes) 000728 1787389100.573288 read 0x016824 thru 0x025159 (0xe936 bytes) 000729 1787389100.589515 trunc from 0x03d6af to 0x035838 (0x7e76 bytes) 000730 1787389100.608775 mapwrite 0x02c828 thru 0x039763 (0xcf3c bytes) 000735 1787389100.731114 trunc from 0x039764 to 0x022a31 (0x16d32 bytes) 000736 1787389100.766991 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22a32): Mirror 10: 0xa7477eba (stale) Mirror 11: 0xa7477eba (stale) Mirror 15: 0xa7477eba (stale) Mirror 20: 0xa7477eba (stale) Mirror 21: 0xa7477eba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22a32): Mirror 10: 0xa7477eba (stale) Mirror 11: 0xa7477eba (stale) Mirror 15: 0xa7477eba (stale) Mirror 20: 0xa7477eba (stale) Mirror 21: 0xa7477eba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22a32: skipped 000737 1787389101.318276 trunc from 0x022a32 to 0x01af09 (0x7b28 bytes) 000738 1787389101.349265 mapread 0x004153 thru 0x009017 (0x4ec5 bytes) 000739 1787389101.379764 write 0x028acf thru 0x035bfc (0xd12e bytes) 000740 1787389101.381431 trunc from 0x035bfd to 0x036eee (0x12f2 bytes) 000741 1787389101.404306 mapwrite 0x034b45 thru 0x03f5e3 (0xaa9f bytes) 000742 1787389101.592125 trunc from 0x03f5e4 to 0x0216bd (0x1df26 bytes) 000743 1787389101.614607 mapwrite 0x00c41b thru 0x00ebb1 (0x2797 bytes) 000744 1787389101.677453 write 0x032f13 thru 0x035f91 (0x307f bytes) 000746 1787389101.689351 write 0x02829f thru 0x029872 (0x15d4 bytes) 000747 1787389101.709442 mapread 0x033bbd thru 0x035f91 (0x23d5 bytes) 000748 1787389101.712401 read 0x02f4e0 thru 0x035f91 (0x6ab2 bytes) 000750 1787389101.723310 mapwrite 0x01825d thru 0x02620b (0xdfaf bytes) 000751 1787389101.830289 write 0x00408e thru 0x00f14c (0xb0bf bytes) 000752 1787389101.837052 trunc from 0x035f92 to 0x00cc0c (0x29385 bytes) 000754 1787389101.871724 mapread 0x00bf89 thru 0x00cc0c (0xc84 bytes) 000755 1787389101.886998 MIRROR_RESYNC 6 mirrors 000757 1787389103.323382 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) 000759 1787389103.824264 mapwrite 0x010d62 thru 0x01277a (0x1a19 bytes) 000760 1787389103.989398 trunc from 0x01277b to 0x01812c (0x59b2 bytes) 000761 1787389104.021213 mapwrite 0x019887 thru 0x01e3d6 (0x4b50 bytes) 000762 1787389104.119985 write 0x024710 thru 0x033000 (0xe8f1 bytes) 000764 1787389104.148343 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33001): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 15: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33001): Mirror 10: 0x1b6c8383 (stale) Mirror 11: 0x1b6c8383 (stale) Mirror 15: 0x1b6c8383 (stale) Mirror 20: 0x1b6c8383 (stale) CRC-32 checksum value for chunk [0, 0x33001): Mirror 1: 0xc80e3a8e Mirror 21: 0xc80e3a8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33001: skipped 000766 1787389105.319766 read 0x0239e8 thru 0x02f75a (0xbd73 bytes) 000768 1787389105.375967 MIRROR_RESYNC 6 mirrors 000769 1787389106.368710 trunc from 0x033001 to 0x036ef5 (0x3ef5 bytes) 000770 1787389106.607185 read 0x0110d8 thru 0x017bc6 (0x6aef bytes) 000774 1787389106.629129 MIRROR_RESYNC 6 mirrors 000775 1787389107.873367 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) 000776 1787389108.268730 write 0x03025e thru 0x032e46 (0x2be9 bytes) 000778 1787389108.394039 trunc from 0x036ef6 to 0x01de13 (0x190e2 bytes) 000780 1787389108.428525 write 0x004dde thru 0x0107df (0xba02 bytes) 000781 1787389108.442826 read 0x00a0a5 thru 0x00f188 (0x50e4 bytes) 000782 1787389108.443984 trunc from 0x01de14 to 0x03d3f6 (0x1f5e3 bytes) 000783 1787389108.466649 mapread 0x03ccef thru 0x03d3f6 (0x708 bytes) 000784 1787389108.482019 mapread 0x016049 thru 0x01b45d (0x5415 bytes) 000786 1787389108.495696 mapread 0x00f08e thru 0x0164ac (0x741f bytes) 000788 1787389108.499635 trunc from 0x03d3f7 to 0x02ac55 (0x127a1 bytes) 000789 1787389108.533200 read 0x0149a2 thru 0x015f65 (0x15c4 bytes) 000790 1787389108.536164 write 0x007da9 thru 0x0110b7 (0x930f bytes) 000792 1787389108.540820 write 0x0391a9 thru 0x03ffff (0x6e57 bytes) 000793 1787389108.553443 MIRROR_RESYNC 5 mirrors 000794 1787389109.734590 mapwrite 0x0328e9 thru 0x039640 (0x6d58 bytes) 000795 1787389110.003124 write 0x031e44 thru 0x03f5c7 (0xd784 bytes) 000796 1787389110.031636 trunc from 0x040000 to 0x031670 (0xe98f bytes) 000797 1787389110.086470 mapwrite 0x00cd5c thru 0x00ff5c (0x3201 bytes) 000799 1787389110.237361 write 0x019f0b thru 0x020e5a (0x6f50 bytes) 000800 1787389110.265457 read 0x02eb88 thru 0x031670 (0x2ae9 bytes) 000804 1787389110.287469 trunc from 0x031671 to 0x007dfa (0x29876 bytes) 000805 1787389110.334606 trunc from 0x007dfb to 0x02ca81 (0x24c87 bytes) 000807 1787389110.369753 mapread 0x00f1e5 thru 0x0117d8 (0x25f4 bytes) 000808 1787389110.398284 mapread 0x0071ec thru 0x007850 (0x665 bytes) 000809 1787389110.399353 mapread 0x0064e7 thru 0x0151b1 (0xeccb bytes) 000810 1787389110.404799 read 0x01eb44 thru 0x0207c3 (0x1c80 bytes) 000811 1787389110.430346 trunc from 0x02ca82 to 0x016c3f (0x15e42 bytes) 000812 1787389110.481204 trunc from 0x016c40 to 0x005883 (0x113bc bytes) 000813 1787389110.511725 mapread 0x002a10 thru 0x005883 (0x2e74 bytes) 000814 1787389110.512936 trunc from 0x005884 to 0x026112 (0x2088f bytes) 000815 1787389110.553251 MIRROR_RESYNC 5 mirrors 000819 1787389112.054995 trunc from 0x026113 to 0x024c39 (0x14d9 bytes) 000822 1787389112.204671 mapread 0x0193f9 thru 0x024c39 (0xb841 bytes) 000824 1787389112.261183 write 0x0241ce thru 0x03384a (0xf67d bytes) 000825 1787389112.291045 MIRROR_RESYNC 5 mirrors 000826 1787389113.936261 trunc from 0x03384b to 0x03439f (0xb55 bytes) 000828 1787389114.102487 read 0x004e29 thru 0x00edbd (0x9f95 bytes) 000830 1787389114.120174 mapread 0x027be9 thru 0x028cc9 (0x10e1 bytes) 000831 1787389114.148280 mapread 0x0256d1 thru 0x02bbca (0x64fa bytes) 000832 1787389114.185591 mapwrite 0x028ad0 thru 0x02bf59 (0x348a bytes) 000833 1787389114.274650 write 0x00c16d thru 0x016306 (0xa19a bytes) 000836 1787389114.304098 read 0x028aaf thru 0x03439f (0xb8f1 bytes) 000837 1787389114.342038 trunc from 0x0343a0 to 0x027be7 (0xc7b8 bytes) 000839 1787389114.373143 read 0x0173f7 thru 0x01e8ce (0x74d8 bytes) 000840 1787389114.374443 write 0x029775 thru 0x02ea25 (0x52b1 bytes) 000841 1787389114.376691 mapread 0x025069 thru 0x02ea25 (0x99bd bytes) 000842 1787389114.383908 trunc from 0x02ea26 to 0x002833 (0x2c1f2 bytes) 000843 1787389114.407039 mapread 0x0000e7 thru 0x002833 (0x274d bytes) 000845 1787389114.409354 trunc from 0x002834 to 0x02bc13 (0x293e0 bytes) 000850 1787389114.435551 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bc14): Mirror 10: 0x7e875763 (stale) Mirror 11: 0x7e875763 (stale) Mirror 20: 0x7e875763 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bc14): Mirror 10: 0x7e875763 (stale) Mirror 11: 0x7e875763 (stale) Mirror 20: 0x7e875763 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bc14: skipped 000851 1787389114.807179 read 0x0074da thru 0x012bc9 (0xb6f0 bytes) 000853 1787389114.829722 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2bc14): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bc14): Mirror 10: 0x7e875763 (stale) Mirror 11: 0x7e875763 (stale) Mirror 20: 0x7e875763 (stale) CRC-32 checksum value for chunk [0, 0x2bc14): Mirror 1: 0xa153164d Mirror 21: 0xa153164d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bc14: skipped 000857 1787389115.915792 mapwrite 0x01a4e3 thru 0x027d53 (0xd871 bytes) 000858 1787389116.130166 trunc from 0x02bc14 to 0x0094e5 (0x2272e bytes) 000859 1787389116.155196 write 0x02f538 thru 0x0352aa (0x5d73 bytes) 000860 1787389116.159888 read 0x02878f thru 0x0336ba (0xaf2c bytes) 000861 1787389116.176800 write 0x02048b thru 0x02fc4a (0xf7c0 bytes) 000863 1787389116.186416 trunc from 0x0352ab to 0x004274 (0x31036 bytes) 000864 1787389116.224100 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x4275): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4275): Mirror 10: 0xd4fcdbc (stale) Mirror 11: 0xd4fcdbc (stale) Mirror 20: 0xd4fcdbc (stale) Mirror 21: 0xb8bed961 (stale) CRC-32 checksum value for chunk [0, 0x4275): Mirror 1: 0xb8bed961 Mirror 22: 0xb8bed961 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4275: skipped 000869 1787389117.062918 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x4275): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4275): Mirror 10: 0xd4fcdbc (stale) Mirror 11: 0xd4fcdbc (stale) Mirror 20: 0xd4fcdbc (stale) Mirror 21: 0xb8bed961 (stale) CRC-32 checksum value for chunk [0, 0x4275): Mirror 1: 0xb8bed961 Mirror 22: 0xb8bed961 Mirror 23: 0xb8bed961 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4275: skipped 000871 1787389118.093940 MIRROR_RESYNC 7 mirrors 000875 1787389119.242001 trunc from 0x004275 to 0x011900 (0xd68c bytes) 000876 1787389119.449960 read 0x007f54 thru 0x011900 (0x99ad bytes) 000877 1787389119.464402 read 0x00235c thru 0x00bb26 (0x97cb bytes) 000878 1787389119.475374 read 0x00b31a thru 0x00f20e (0x3ef5 bytes) 000879 1787389119.476054 trunc from 0x011901 to 0x03e530 (0x2cc30 bytes) 000880 1787389119.499569 mapwrite 0x015f0e thru 0x0212a1 (0xb394 bytes) 000881 1787389119.578909 trunc from 0x03e531 to 0x01b963 (0x22bcd bytes) 000882 1787389119.605636 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b964): Mirror 11: 0xfa48b4b (stale) Mirror 20: 0xfa48b4b (stale) Mirror 21: 0xfa48b4b (stale) Mirror 22: 0xfa48b4b (stale) Mirror 23: 0xfa48b4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b964): Mirror 11: 0xfa48b4b (stale) Mirror 20: 0xfa48b4b (stale) Mirror 21: 0xfa48b4b (stale) Mirror 22: 0xfa48b4b (stale) Mirror 23: 0xfa48b4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b964: skipped 000883 1787389120.048409 MIRROR_RESYNC 6 mirrors 000884 1787389121.717718 read 0x016ba1 thru 0x01b963 (0x4dc3 bytes) 000885 1787389121.739089 write 0x024ae9 thru 0x030e19 (0xc331 bytes) 000886 1787389121.929326 write 0x0206ff thru 0x029c4e (0x9550 bytes) 000888 1787389121.941720 write 0x00aee6 thru 0x015f0e (0xb029 bytes) 000891 1787389121.956311 mapwrite 0x0321b9 thru 0x03ffff (0xde47 bytes) 000892 1787389122.086820 trunc from 0x040000 to 0x0098d3 (0x3672c bytes) 000897 1787389122.133551 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x98d4): Mirror 11: 0x5d5b4c97 (stale) Mirror 20: 0x5d5b4c97 (stale) Mirror 21: 0x5d5b4c97 (stale) Mirror 23: 0x5d5b4c97 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x98d4): Mirror 11: 0x5d5b4c97 (stale) Mirror 20: 0x5d5b4c97 (stale) Mirror 21: 0x5d5b4c97 (stale) Mirror 23: 0x5d5b4c97 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x98d4: skipped 000900 1787389122.637032 trunc from 0x0098d4 to 0x02f412 (0x25b3f bytes) 000901 1787389122.665120 trunc from 0x02f413 to 0x0284fc (0x6f16 bytes) 000902 1787389122.687328 mapwrite 0x036c31 thru 0x036f16 (0x2e6 bytes) 000905 1787389122.767569 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36f17): Mirror 11: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f17): Mirror 11: 0xff71f23f (stale) Mirror 20: 0xff71f23f (stale) Mirror 21: 0xff71f23f (stale) Mirror 23: 0xff71f23f (stale) CRC-32 checksum value for chunk [0, 0x36f17): Mirror 1: 0xa020ebd5 Mirror 24: 0xa020ebd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f17: skipped 000906 1787389123.992746 trunc from 0x036f17 to 0x0096be (0x2d858 bytes) 000908 1787389124.152753 trunc from 0x0096bf to 0x003c17 (0x5aa7 bytes) 000910 1787389124.187904 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c18): Mirror 11: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c18): Mirror 11: 0xbd95f85f (stale) Mirror 20: 0xbd95f85f (stale) Mirror 21: 0xbd95f85f (stale) Mirror 23: 0xbd95f85f (stale) Mirror 24: 0xbd95f85f (stale) CRC-32 checksum value for chunk [0, 0x3c18): Mirror 1: 0xbd95f85f Mirror 25: 0xbd95f85f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c18: skipped 000911 1787389125.079758 read 0x000295 thru 0x0032f6 (0x3062 bytes) 000912 1787389125.093363 write 0x03978f thru 0x03ffff (0x6871 bytes) 000914 1787389125.182684 mapwrite 0x01cf4b thru 0x024aa8 (0x7b5e bytes) 000915 1787389125.293489 trunc from 0x040000 to 0x00dc19 (0x323e6 bytes) 000916 1787389125.325580 read 0x007c0c thru 0x00dc19 (0x600e bytes) 000917 1787389125.347646 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdc1a): Mirror 20: 0x5f0a4997 (stale) Mirror 21: 0x5f0a4997 (stale) Mirror 23: 0x5f0a4997 (stale) Mirror 24: 0x5f0a4997 (stale) Mirror 25: 0x5f0a4997 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdc1a): Mirror 20: 0x5f0a4997 (stale) Mirror 21: 0x5f0a4997 (stale) Mirror 23: 0x5f0a4997 (stale) Mirror 24: 0x5f0a4997 (stale) Mirror 25: 0x5f0a4997 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdc1a: skipped 000918 1787389125.864504 mapread 0x001521 thru 0x002aae (0x158e bytes) 000919 1787389125.886775 MIRROR_RESYNC 6 mirrors 000920 1787389127.145501 mapread 0x0049b1 thru 0x00dc19 (0x9269 bytes) 000921 1787389127.170646 write 0x002db1 thru 0x00a0ad (0x72fd bytes) 000922 1787389127.271231 trunc from 0x00dc1a to 0x02a011 (0x1c3f8 bytes) 000923 1787389127.298016 mapwrite 0x016d85 thru 0x0192bd (0x2539 bytes) 000926 1787389127.395021 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a012): Mirror 21: 0xf790eefe (stale) Mirror 23: 0xf790eefe (stale) Mirror 24: 0xf790eefe (stale) Mirror 25: 0xf790eefe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a012): Mirror 21: 0xf790eefe (stale) Mirror 23: 0xf790eefe (stale) Mirror 24: 0xf790eefe (stale) Mirror 25: 0xf790eefe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a012: skipped 000927 1787389127.774045 write 0x01b0de thru 0x023751 (0x8674 bytes) 000929 1787389127.793939 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a012): Mirror 23: 0xf790eefe (stale) Mirror 24: 0xf790eefe (stale) Mirror 25: 0xf790eefe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a012): Mirror 23: 0xf790eefe (stale) Mirror 24: 0xf790eefe (stale) Mirror 25: 0xf790eefe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a012: skipped 000933 1787389128.217862 write 0x01155b thru 0x01a337 (0x8ddd bytes) 000937 1787389128.239323 read 0x02019d thru 0x0264b0 (0x6314 bytes) 000938 1787389128.249748 write 0x038303 thru 0x03ffff (0x7cfd bytes) 000940 1787389128.253382 mapread 0x00abdd thru 0x012463 (0x7887 bytes) 000941 1787389128.285797 MIRROR_RESYNC 4 mirrors 000943 1787389129.170436 MIRROR_EXTEND to 5 mirrors 000944 1787389130.178297 write 0x03a9bd thru 0x03ffff (0x5643 bytes) 000947 1787389130.322438 write 0x03f30a thru 0x03ffff (0xcf6 bytes) 000948 1787389130.323831 read 0x03579b thru 0x03ef36 (0x979c bytes) 000949 1787389130.348992 trunc from 0x040000 to 0x02b876 (0x14789 bytes) 000950 1787389130.441183 mapwrite 0x004fd9 thru 0x014af2 (0xfb1a bytes) 000951 1787389130.596729 read 0x018b8b thru 0x02495b (0xbdd1 bytes) 000952 1787389130.615629 mapread 0x0297c8 thru 0x02b876 (0x20af bytes) 000955 1787389130.619349 mapread 0x00dbbd thru 0x010f76 (0x33ba bytes) 000956 1787389130.620764 mapwrite 0x03307c thru 0x035fab (0x2f30 bytes) 000958 1787389130.695310 trunc from 0x035fac to 0x01a269 (0x1bd42 bytes) 000960 1787389130.716631 trunc from 0x01a26a to 0x0381e5 (0x1df7c bytes) 000962 1787389130.742461 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x140:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x381e6): Mirror 23: 0xa3a87726 (stale) Mirror 24: 0xa3a87726 (stale) Mirror 26: 0xa3a87726 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381e6): Mirror 23: 0xa3a87726 (stale) Mirror 24: 0xa3a87726 (stale) Mirror 26: 0xa3a87726 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381e6: skipped 000966 1787389131.312555 mapread 0x0241b9 thru 0x02cf64 (0x8dac bytes) 000971 1787389131.357630 mapread 0x00cd77 thru 0x0135a3 (0x682d bytes) 000972 1787389131.366749 mapwrite 0x01ed4f thru 0x027ea6 (0x9158 bytes) 000976 1787389131.430955 mapwrite 0x0379d8 thru 0x03966e (0x1c97 bytes) 000978 1787389131.558636 read 0x02acc0 thru 0x03329b (0x85dc bytes) 000980 1787389131.567944 read 0x0359b8 thru 0x037bdd (0x2226 bytes) 000981 1787389131.568882 mapread 0x019b87 thru 0x01eb9e (0x5018 bytes) 000983 1787389131.599714 MIRROR_RESYNC 4 mirrors 000984 1787389132.530424 trunc from 0x03966f to 0x0193db (0x20293 bytes) 000985 1787389132.623553 trunc from 0x0193dc to 0x01ddc1 (0x49e6 bytes) 000990 1787389132.665378 write 0x0276f8 thru 0x035dc1 (0xe6ca bytes) 000993 1787389132.675197 trunc from 0x035dc2 to 0x0122fd (0x23ac4 bytes) 000994 1787389132.708931 read 0x00a987 thru 0x00e390 (0x3a0a bytes) 000995 1787389132.720867 mapwrite 0x02db66 thru 0x03cca7 (0xf142 bytes) 000996 1787389132.834151 trunc from 0x03cca8 to 0x004d8c (0x37f1b bytes) 000998 1787389132.908085 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x4d8d): Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4d8d): Mirror 23: 0xa45fd2eb (stale) Mirror 24: 0xa45fd2eb (stale) Mirror 26: 0xa45fd2eb (stale) CRC-32 checksum value for chunk [0, 0x4d8d): Mirror 1: 0xa45fd2eb Mirror 27: 0xa45fd2eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4d8d: skipped 000999 1787389134.069691 mapread 0x001b00 thru 0x004d8c (0x328d bytes) All operations completed A-OK! PASS 70a (149s) == sanity-flr test 71: check mirror extend parent fid ==== 04:59:00 (1787389140) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 04:59:03 (1787389143) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.300969 s, 7.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x281:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173092 s, 6.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x281:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x265:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x281:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x265:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x264:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x282:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 04:59:14 (1787389154) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre2 Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3280512 .. Extending file size to 6491520 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6691744 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8242752 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7735424 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9220800 .. Extending file size to 8552192 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56745 56746 56747 56750 56751 done 192.168.204.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre2 (opts:) 192.168.204.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (92s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 05:00:46 (1787389246) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre2 Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7359680 .. Extending file size to 5393536 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7875872 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8291648 .. Extending file size to 8406528 .. Extending file size to 8449760 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8527872 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8752672 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8594336 .. Waiting 57968 57969 57972 57973 57975 done 192.168.204.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre2 (opts:) 192.168.204.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 05:02:13 (1787389333) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105132 s, 19.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.384211 s, 5.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb68587ce (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:02:34 (1787389354) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105337 s, 19.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 05:02:42 (1787389362) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:02:50 (1787389370) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.260231 s, 8.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x298:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x299:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:02:58 (1787389378) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.235377 s, 8.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29b:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:03:06 (1787389386) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.221726 s, 9.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29d:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:03:39 (1787389419) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg423-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222046 s, 4.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x281:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x281:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x282:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:04:42 (1787389482) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg423-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.452873 s, 4.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (66s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:05:47 (1787389547) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg423-server mds-ost sync done. Creating new pool test_204f oleg423-server: Pool lustre.test_204f created Adding targets to pool oleg423-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg423-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.252489 s, 8.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg423-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg423-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg423-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:07:08 (1787389628) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:07:16 (1787389636) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:07:24 (1787389644) oleg423-server: Pool lustre.test_206 created Adding targets to pool oleg423-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg423-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg423-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg423-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg423-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:07:48 (1787389668) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110394 s, 9.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610402 s, 17.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x88bda2d4 Mirror 3: 0x88bda2d4 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:07:59 (1787389679) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:08:03 (1787389683) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:08:06 (1787389686) Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.146639 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x6c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x22c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x22c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x22c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1860000 Read this (4 bytes): BBBB 192.168.204.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg423-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:08:44 (1787389724) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0227582 s, 360 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:08:51 (1787389731) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376309 s, 27.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1b4 lmm_fid: [0x200000404:0x1b4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1b4 lmm_fid: [0x200000404:0x1b4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834356 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 05:08:53.000000000 -0400 Modify: 2026-08-22 05:08:53.000000000 -0400 Change: 2026-08-22 05:08:54.000000000 -0400 Birth: 2026-08-22 05:08:53.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:09:00 (1787389740) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:09:04 (1787389744) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.883587 s, 46.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0862573 s, 47.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 22 05:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b6:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 22 05:09 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:09:14 (1787389754) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:09:22 (1787389762) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1838 sec =========== 05:09:32 (1787389772) === sanity-flr: start cleanup 05:09:34 (1787389774) === === sanity-flr: finish cleanup 05:09:41 (1787389781) ===