-----============= acceptance-small: sanity-flr ============----- Thu Sep 10 13:35:29 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg118-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:35:52 (1789061752) === oleg118-client.virtnet: executing check_config_client /mnt/lustre oleg118-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg118-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff924c51141800.idle_timeout=debug osc.lustre-OST0001-osc-ffff924c51141800.idle_timeout=debug disable quota as required oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:36:33 (1789061793) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7620] == sanity-flr test 0a: lfs mirror create with -N option == 13:36:36 (1789061796) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:36:55 (1789061815) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:37:00 (1789061820) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:37:03 (1789061823) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:37:33 (1789061853) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:37:36 (1789061856) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 13:37:40 (1789061860) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff924c51141800.stats=clear osc.lustre-OST0001-osc-ffff924c51141800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:37:50 (1789061870) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:38:04 (1789061884) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:38:14 (1789061894) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:38:19 (1789061899) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (11s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:38:30 (1789061910) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 13:38:41 (1789061921) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:38:50 (1789061930) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:38:55 (1789061935) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:39:04 (1789061944) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:39:17 (1789061957) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:39:26 (1789061966) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 3.02275 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281746 s, 3.7 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:39:41 (1789061981) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.197493 s, 21.2 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:39:51 (1789061991) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:40:05 (1789062005) Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff924c51141800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff924c51141800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff924c51141800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff924c51141800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg118-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (92s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:41:36 (1789062096) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.755157 s, 15.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff924c51141800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff924c51141800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (54s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:42:31 (1789062151) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff924c51141800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff924c51141800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff924c51141800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff924c51141800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg118-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (89s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:44:01 (1789062241) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.97187 s, 10.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 10 13:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.450s user 0m0.000s sys 0m0.193s Stopping /mnt/lustre-ost1 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff924c51141800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff924c51141800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.534s user 0m0.001s sys 0m0.184s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg118-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg118-server oleg118-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff924c51141800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff924c51141800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.503s user 0m0.000s sys 0m0.185s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg118-server: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg118-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff924c51141800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff924c51141800.ost_server_uuid in FULL state after 0 sec PASS 33b (122s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:46:04 (1789062364) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:46:08 (1789062368) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:46:13 (1789062373) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 13:46:17 (1789062377) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 13:46:28 (1789062388) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.8721 s, 14.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:46:45 (1789062405) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441944 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040088 s, 26.2 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:46:54 (1789062414) PASS 36c (11s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:47:05 (1789062425) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00095742 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00085402 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127188 s, 805 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00529001 s, 194 kB/s flv=10 olv=10 PASS 36d (15s) == sanity-flr test 37: mirror I/O API verification ======= 13:47:21 (1789062441) create test files with size 15733375 .. b9e79b93dc6a699eea4bb305ec48ef55 - 94fcd9c308fbd91e044dd93a7d607ceb - a554785ed2a5665e9d02b3f15caa0fcb - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg118-client.virtnet /mnt/lustre (opts:) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre PASS 37 (35s) == sanity-flr test 38: resync ============================ 13:47:56 (1789062476) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (24s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:48:20 (1789062500) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:48:29 (1789062509) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160728 s, 6.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399276 s, 26.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (13s) == sanity-flr test 41: lfs mirror resync check =========== 13:48:42 (1789062522) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0992598 s, 21.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.604575 s, 6.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246614 s, 8.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (24s) == sanity-flr test 42: lfs mirror verify ================= 13:49:06 (1789062546) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 13:49:11 (1789062551) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:49:14 (1789062554) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880799 s, 11.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 13:49:25 (1789062565) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:49:29 (1789062569) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:49:42 (1789062582) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.372796 s, 28.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg118-client.virtnet /mnt/lustre (opts:) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:49:54 (1789062594) llite.lustre-ffff924c6041a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.363816 s, 28.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg118-client.virtnet /mnt/lustre (opts:) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff924c43b79800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:50:13 (1789062613) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (56s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:51:08 (1789062668) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:51:52 (1789062712) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 13:51:57 (1789062717) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.274177 s, 11.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.265985 s, 11.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5ed00549 (nosync,stale) Mirror 2: 0x5ed00549 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5d28b8a67baf1711bada76fa3c7651e2 - mirror 1 checksum: 1f90133a11a78932cc8de3b214d5bae4 - mirror 2 checksum: 5d28b8a67baf1711bada76fa3c7651e2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T13:52:00-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T13:52:00-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5d28b8a67baf1711bada76fa3c7651e2 - mirror 1 checksum: 1f90133a11a78932cc8de3b214d5bae4 - mirror 2 checksum: 1f90133a11a78932cc8de3b214d5bae4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T13:52:00-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T13:52:00-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (15s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:52:15 (1789062735) Failing mds1 on oleg118-server Stopping /mnt/lustre-mds1 (opts:) on oleg118-server 13:52:25 (1789062745) shut down facet: mds1 facet_host: oleg118-server facet_failover_host: oleg118-server Failover mds1 to oleg118-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 13:53:00 (1789062780) targets are mounted 13:53:00 (1789062780) facet_failover done oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649564 s, 16.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:53:19 (1789062799) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:53:23 (1789062803) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:53:28 (1789062808) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0349243 s, 469 kB/s PASS 50c (11s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:53:38 (1789062818) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:53:43 (1789062823) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0208056 s, 197 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.1478 s, 27.7 kB/s PASS 60a (12s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:53:55 (1789062835) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (41s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:54:36 (1789062876) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (33s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:55:09 (1789062909) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:55:36 (1789062936) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:56:02 (1789062962) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 13:56:12 (1789062972) mirror create pid 54350 mirror split pid 54352 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54350 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54352 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (71s) == sanity-flr test 70a: flr mode fsx test ================ 13:57:23 (1789063043) Seed set to 3046 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1789063046.755916 write 0x019802 thru 0x021eb9 (0x86b8 bytes) truncating to largest ever: 0x2df0e 000004 1789063046.767326 trunc from 0x021eba to 0x02df0d (0xc054 bytes) truncating to largest ever: 0x2e5a4 000005 1789063046.820412 trunc from 0x02df0e to 0x02e5a3 (0x696 bytes) 000006 1789063046.851587 punch from 0x0165a8 to 0x01e18d (0x7be6 bytes) truncating to largest ever: 0x384d6 000007 1789063046.875653 trunc from 0x02e5a4 to 0x0384d5 (0x9f32 bytes) 000008 1789063046.955264 MIRROR_EXTEND to 2 mirrors 000009 1789063049.404591 mapread 0x025ddf thru 0x0268fd (0xb1f bytes) 000010 1789063049.475118 mapread 0x0198a4 thru 0x0262ab (0xca08 bytes) 000011 1789063049.589638 mapread 0x01a117 thru 0x01d6e5 (0x35cf bytes) 000013 1789063049.666957 trunc from 0x0384d6 to 0x022c3f (0x15896 bytes) 000014 1789063049.879467 write 0x0004b4 thru 0x008d19 (0x8866 bytes) 000021 1789063049.927530 write 0x03a76a thru 0x03db1d (0x33b4 bytes) 000023 1789063049.929402 write 0x00a7cb thru 0x018457 (0xdc8d bytes) 000026 1789063049.988141 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: 0x67fa9d2c (stale) CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 1: 0x21de2e4a Mirror 3: 0x21de2e4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1e: skipped 000027 1789063052.385413 read 0x00bfb8 thru 0x00ed5e (0x2da7 bytes) 000028 1789063052.426916 punch from 0x023b91 to 0x024def (0x125f bytes) 000029 1789063052.645794 read 0x003880 thru 0x010786 (0xcf07 bytes) 000030 1789063052.667246 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: 0x67fa9d2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: 0x67fa9d2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1e: skipped 000031 1789063053.325266 mapread 0x00a78a thru 0x00e8cb (0x4142 bytes) 000032 1789063053.371090 read 0x020d33 thru 0x02d8c0 (0xcb8e bytes) 000033 1789063053.404735 mapread 0x010adf thru 0x01641d (0x593f bytes) 000034 1789063053.476463 mapwrite 0x013009 thru 0x01c126 (0x911e bytes) 000035 1789063053.651166 read 0x02a3e1 thru 0x02ca4f (0x266f bytes) 000039 1789063053.668159 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: 0x67fa9d2c (stale) CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 1: 0x333a83f1 Mirror 3: 0x333a83f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1e: skipped 000040 1789063056.037466 punch from 0x00471f to 0x013bdb (0xf4bd bytes) 000041 1789063056.191239 mapwrite 0x0130b4 thru 0x0213d6 (0xe323 bytes) 000042 1789063056.341506 mapread 0x0048e9 thru 0x00f2c7 (0xa9df bytes) 000043 1789063056.360196 punch from 0x03b448 to 0x03db1d (0x26d6 bytes) 000045 1789063056.368334 punch from 0x037381 to 0x03db1d (0x679d bytes) 000046 1789063056.375141 mapwrite 0x036c78 thru 0x03daa5 (0x6e2e bytes) 000047 1789063056.662792 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 2: 0x67fa9d2c (stale) Mirror 3: 0x333a83f1 (stale) CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 1: 0x4132e43d Mirror 4: 0x4132e43d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1e: skipped 000048 1789063058.810499 punch from 0x02a8e7 to 0x02d6c9 (0x2de3 bytes) 000049 1789063059.043192 mapwrite 0x00b7f4 thru 0x019a35 (0xe242 bytes) 000050 1789063059.270094 mapwrite 0x003953 thru 0x010c44 (0xd2f2 bytes) 000051 1789063059.415641 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 3: 0x333a83f1 (stale) Mirror 4: 0x4132e43d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db1e): Mirror 3: 0x333a83f1 (stale) Mirror 4: 0x4132e43d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db1e: skipped 000052 1789063060.131890 mapread 0x01f631 thru 0x02ba2f (0xc3ff bytes) 000053 1789063060.195940 MIRROR_RESYNC 3 mirrors 000055 1789063061.589099 punch from 0x021178 to 0x02dec6 (0xcd4f bytes) truncating to largest ever: 0x3bd0f 000057 1789063061.693141 trunc from 0x03db1e to 0x03bd0e (0x1e0f bytes) 000062 1789063061.738207 mapread 0x01ab0e thru 0x028f68 (0xe45b bytes) 000064 1789063061.804754 punch from 0x00269c to 0x01038e (0xdcf3 bytes) 000066 1789063061.814813 trunc from 0x03bd0f to 0x021358 (0x1a9b6 bytes) 000067 1789063061.843720 mapwrite 0x035fd1 thru 0x03d010 (0x7040 bytes) 000068 1789063061.949994 write 0x00e548 thru 0x01449b (0x5f54 bytes) 000069 1789063061.967317 punch from 0x0124bc to 0x01cafb (0xa640 bytes) 000070 1789063061.987649 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d011): Mirror 4: 0x281baf4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d011): Mirror 4: 0x281baf4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d011: skipped 000071 1789063062.536138 trunc from 0x03d011 to 0x00e51d (0x2eaf3 bytes) 000072 1789063062.565856 mapwrite 0x036648 thru 0x03ffff (0x99b8 bytes) 000073 1789063062.889363 trunc from 0x040000 to 0x008797 (0x37868 bytes) 000074 1789063062.943782 punch from 0x0018ca to 0x008797 (0x6ece bytes) 000075 1789063062.956644 trunc from 0x008798 to 0x031c32 (0x2949b bytes) 000076 1789063063.012761 punch from 0x013fc4 to 0x02135c (0xd399 bytes) 000078 1789063063.036155 read 0x031391 thru 0x031c32 (0x8a2 bytes) 000079 1789063063.054167 mapwrite 0x002809 thru 0x006074 (0x386c bytes) 000081 1789063063.304700 trunc from 0x031c33 to 0x00cf80 (0x24cb2 bytes) 000082 1789063063.380389 MIRROR_RESYNC 2 mirrors 000083 1789063064.632341 read 0x008c8a thru 0x00a3a8 (0x171f bytes) 000084 1789063064.659474 punch from 0x004d0d to 0x00cf80 (0x8274 bytes) 000085 1789063064.843513 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xcf81): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcf81): Mirror 4: 0xa1964fc9 (stale) CRC-32 checksum value for chunk [0, 0xcf81): Mirror 1: 0xbf31f131 Mirror 5: 0xbf31f131 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcf81: skipped 000086 1789063066.518646 mapread 0x00204b thru 0x00614c (0x4102 bytes) 000087 1789063066.642157 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) 000088 1789063067.161190 trunc from 0x00cf81 to 0x034ca5 (0x27d25 bytes) 000089 1789063067.308720 read 0x02576a thru 0x02f4c0 (0x9d57 bytes) 000091 1789063067.327559 read 0x014b46 thru 0x0160e7 (0x15a2 bytes) 000095 1789063067.351528 trunc from 0x034ca6 to 0x0155d0 (0x1f6d5 bytes) 000096 1789063067.427822 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x155d1): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x155d1): Mirror 5: 0x84c9a009 (stale) CRC-32 checksum value for chunk [0, 0x155d1): Mirror 1: 0x84c9a009 Mirror 6: 0x84c9a009 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x155d1: skipped 000097 1789063069.167482 trunc from 0x0155d1 to 0x020177 (0xaba7 bytes) 000098 1789063069.394238 write 0x0387dd thru 0x03a72a (0x1f4e bytes) 000099 1789063069.397446 mapread 0x00fee1 thru 0x015c17 (0x5d37 bytes) 000100 1789063069.480773 mapread 0x02dce2 thru 0x032eac (0x51cb bytes) 000101 1789063069.506835 read 0x025aa3 thru 0x027108 (0x1666 bytes) 000103 1789063069.531090 mapread 0x022e52 thru 0x03209d (0xf24c bytes) 000104 1789063069.554077 mapwrite 0x03f15b thru 0x03ffff (0xea5 bytes) 000108 1789063069.813616 mapwrite 0x01d9ba thru 0x01ff10 (0x2557 bytes) 000109 1789063070.007062 trunc from 0x040000 to 0x00ff09 (0x300f6 bytes) 000110 1789063070.081580 write 0x003b18 thru 0x007b29 (0x4012 bytes) 000111 1789063070.121776 mapwrite 0x034755 thru 0x03ffff (0xb8ab bytes) 000112 1789063070.326556 MIRROR_RESYNC 3 mirrors 000113 1789063072.137283 punch from 0x033f08 to 0x0364ea (0x25e3 bytes) 000114 1789063072.255019 MIRROR_RESYNC 3 mirrors 000115 1789063074.041040 MIRROR_RESYNC 3 mirrors 000116 1789063074.943631 trunc from 0x040000 to 0x02eeec (0x11113 bytes) 000117 1789063075.070760 read 0x0048c2 thru 0x005303 (0xa42 bytes) 000118 1789063075.082866 mapread 0x02c830 thru 0x02eeec (0x26bd bytes) 000119 1789063075.111496 mapread 0x0197df thru 0x01d4d7 (0x3cf9 bytes) 000120 1789063075.132588 read 0x0039a4 thru 0x007968 (0x3fc5 bytes) 000122 1789063075.149982 mapwrite 0x006732 thru 0x00e42a (0x7cf9 bytes) 000123 1789063075.279545 read 0x02812e thru 0x02eeec (0x6dbf bytes) 000124 1789063075.280299 trunc from 0x02eeed to 0x0372ff (0x8413 bytes) 000125 1789063075.347453 read 0x033461 thru 0x0372ff (0x3e9f bytes) 000126 1789063075.364613 punch from 0x005ed3 to 0x009db8 (0x3ee6 bytes) 000127 1789063075.377700 mapread 0x02de3e thru 0x0372ff (0x94c2 bytes) 000128 1789063075.386403 read 0x02bb4e thru 0x0372ff (0xb7b2 bytes) 000129 1789063075.387283 trunc from 0x037300 to 0x00bfae (0x2b351 bytes) 000130 1789063075.425747 mapread 0x009e64 thru 0x00bfae (0x214b bytes) 000131 1789063075.439530 read 0x0045d3 thru 0x00bfae (0x79dc bytes) 000133 1789063075.442683 mapwrite 0x031cd0 thru 0x03afc9 (0x92fa bytes) 000135 1789063075.641581 mapwrite 0x0082c5 thru 0x0129f9 (0xa735 bytes) 000138 1789063075.807704 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3afca): Mirror 6: 0xd29c344a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afca): Mirror 6: 0xd29c344a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afca: skipped 000140 1789063076.341113 mapread 0x025467 thru 0x028abd (0x3657 bytes) 000141 1789063076.388095 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3afca): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afca): Mirror 6: 0xd29c344a (stale) CRC-32 checksum value for chunk [0, 0x3afca): Mirror 1: 0x2bc3dcd2 Mirror 7: 0x2bc3dcd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afca: skipped 000142 1789063078.172011 write 0x01fc49 thru 0x02b349 (0xb701 bytes) 000146 1789063078.313103 mapwrite 0x01866a thru 0x01b8d1 (0x3268 bytes) 000148 1789063078.420709 write 0x01d5d6 thru 0x02b028 (0xda53 bytes) 000149 1789063078.422065 read 0x0376ec thru 0x03afc9 (0x38de bytes) 000150 1789063078.433689 mapwrite 0x02fef9 thru 0x039457 (0x955f bytes) 000151 1789063078.531641 trunc from 0x03afca to 0x009355 (0x31c74 bytes) truncating to largest ever: 0x3f0cb 000153 1789063078.609913 trunc from 0x009356 to 0x03f0ca (0x35d75 bytes) 000155 1789063078.719819 mapread 0x008ba6 thru 0x010698 (0x7af3 bytes) 000156 1789063078.754219 read 0x032660 thru 0x03aa52 (0x83f3 bytes) 000157 1789063078.772576 write 0x0347b0 thru 0x03ffff (0xb850 bytes) 000158 1789063078.793060 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8f8d4afc (stale) Mirror 7: 0x64b4f9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x39c99b36 Mirror 8: 0x39c99b36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000159 1789063081.476646 punch from 0x02fda7 to 0x03ac19 (0xae73 bytes) 000160 1789063081.686270 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8f8d4afc (stale) Mirror 8: 0x39c99b36 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8f8d4afc (stale) Mirror 8: 0x39c99b36 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000161 1789063082.495544 write 0x018b16 thru 0x01aada (0x1fc5 bytes) 000162 1789063082.543387 mapwrite 0x007166 thru 0x00a421 (0x32bc bytes) 000164 1789063082.876036 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8f8d4afc (stale) Mirror 8: 0x39c99b36 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd6b1a2f3 Mirror 9: 0xd6b1a2f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000165 1789063085.973033 punch from 0x01b906 to 0x01eca1 (0x339c bytes) 000168 1789063086.096052 MIRROR_RESYNC 4 mirrors 000169 1789063089.015062 write 0x03a9a2 thru 0x03ffff (0x565e bytes) 000170 1789063089.268751 read 0x03c368 thru 0x03e232 (0x1ecb bytes) 000171 1789063089.271570 punch from 0x03523d to 0x03ffff (0xadc3 bytes) 000172 1789063089.301314 read 0x01b012 thru 0x021b33 (0x6b22 bytes) 000173 1789063089.369344 mapwrite 0x02a210 thru 0x0317c9 (0x75ba bytes) 000174 1789063089.639764 trunc from 0x040000 to 0x03105a (0xefa5 bytes) 000175 1789063089.723205 trunc from 0x03105b to 0x02b336 (0x5d24 bytes) 000177 1789063089.796690 mapread 0x01d11f thru 0x02538f (0x8271 bytes) 000178 1789063089.797956 trunc from 0x02b337 to 0x0167c1 (0x14b75 bytes) truncating to largest ever: 0x3f787 000182 1789063089.885778 trunc from 0x0167c2 to 0x03f786 (0x28fc5 bytes) 000183 1789063089.928063 mapread 0x0079b4 thru 0x00868a (0xcd7 bytes) 000184 1789063089.933180 mapwrite 0x01b135 thru 0x02a1fe (0xf0ca bytes) 000185 1789063090.338290 write 0x01bf54 thru 0x01fa17 (0x3ac4 bytes) 000187 1789063090.339839 read 0x016c41 thru 0x02395d (0xcd1d bytes) 000191 1789063090.375277 mapwrite 0x011d7d thru 0x013868 (0x1aec bytes) 000193 1789063090.566367 trunc from 0x03f787 to 0x00f9ef (0x2fd97 bytes) 000194 1789063090.668574 mapwrite 0x006956 thru 0x010f57 (0xa602 bytes) 000195 1789063090.928035 trunc from 0x010f58 to 0x01d8e3 (0xc98c bytes) 000196 1789063090.974554 punch from 0x01ca53 to 0x01d8e3 (0xe91 bytes) 000198 1789063091.000219 punch from 0x004959 to 0x011ca7 (0xd34f bytes) 000200 1789063091.016612 write 0x010cff thru 0x014052 (0x3354 bytes) 000201 1789063091.065793 punch from 0x004bc8 to 0x013c94 (0xf0cd bytes) 000202 1789063091.122294 write 0x00e56d thru 0x011660 (0x30f4 bytes) 000203 1789063091.139343 write 0x02d521 thru 0x031135 (0x3c15 bytes) 000209 1789063091.146746 mapwrite 0x001da4 thru 0x00b794 (0x99f1 bytes) 000210 1789063091.519704 mapread 0x008856 thru 0x016a90 (0xe23b bytes) 000211 1789063091.664855 mapread 0x00d783 thru 0x013a79 (0x62f7 bytes) 000214 1789063091.682384 write 0x00c2f1 thru 0x0193fb (0xd10b bytes) 000216 1789063091.740449 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31136): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31136): Mirror 6: 0x9d024a42 (stale) Mirror 8: 0x9d024a42 (stale) Mirror 9: 0x9d024a42 (stale) CRC-32 checksum value for chunk [0, 0x31136): Mirror 1: 0x9cd3b9a0 Mirror 10: 0x9cd3b9a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31136: skipped 000218 1789063094.240729 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31136): Mirror 8: 0x9d024a42 (stale) Mirror 9: 0x9d024a42 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31136): Mirror 8: 0x9d024a42 (stale) Mirror 9: 0x9d024a42 (stale) CRC-32 checksum value for chunk [0, 0x31136): Mirror 1: 0x9cd3b9a0 Mirror 10: 0x9cd3b9a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31136: skipped 000220 1789063095.124262 MIRROR_RESYNC 4 mirrors 000222 1789063097.086593 mapwrite 0x002396 thru 0x005c9a (0x3905 bytes) 000223 1789063097.455619 mapwrite 0x01e4ee thru 0x0269e1 (0x84f4 bytes) 000225 1789063097.546822 mapwrite 0x008038 thru 0x00f527 (0x74f0 bytes) 000226 1789063097.612873 MIRROR_RESYNC 4 mirrors 000227 1789063098.914460 read 0x00d371 thru 0x015822 (0x84b2 bytes) 000228 1789063098.930571 write 0x03bed6 thru 0x03ffff (0x412a bytes) 000229 1789063099.045855 mapwrite 0x03595d thru 0x03ffff (0xa6a3 bytes) 000230 1789063099.153322 mapwrite 0x017eb9 thru 0x023d0c (0xbe54 bytes) 000231 1789063099.341687 write 0x028877 thru 0x037c8f (0xf419 bytes) 000232 1789063099.343108 mapread 0x02714a thru 0x0363c5 (0xf27c bytes) 000233 1789063099.345691 mapread 0x02b169 thru 0x02b3c0 (0x258 bytes) 000234 1789063099.352894 MIRROR_RESYNC 4 mirrors 000236 1789063100.362789 read 0x0234b1 thru 0x02afa1 (0x7af1 bytes) 000237 1789063100.375285 read 0x029cbc thru 0x03923c (0xf581 bytes) 000238 1789063100.390228 punch from 0x00f671 to 0x018782 (0x9112 bytes) 000240 1789063100.436806 punch from 0x0119cb to 0x01db93 (0xc1c9 bytes) 000241 1789063100.445443 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x7f7f8738 (stale) Mirror 10: 0x7f7f8738 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x7f7f8738 (stale) Mirror 10: 0x7f7f8738 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000242 1789063100.884843 punch from 0x03b18a to 0x03c48f (0x1306 bytes) 000244 1789063100.900113 read 0x002893 thru 0x00b36e (0x8adc bytes) 000245 1789063100.928885 mapwrite 0x01197f thru 0x015546 (0x3bc8 bytes) 000246 1789063101.077893 write 0x013aaa thru 0x01a8f0 (0x6e47 bytes) 000248 1789063101.091935 punch from 0x022f8b to 0x031a1f (0xea95 bytes) 000249 1789063101.133931 mapwrite 0x02f160 thru 0x039c1f (0xaac0 bytes) 000250 1789063101.304234 mapwrite 0x0066cc thru 0x00c762 (0x6097 bytes) 000253 1789063101.435639 trunc from 0x040000 to 0x01706b (0x28f94 bytes) 000256 1789063101.585191 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1706c): Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1706c): Mirror 8: 0x405e3a3c (stale) Mirror 10: 0x405e3a3c (stale) CRC-32 checksum value for chunk [0, 0x1706c): Mirror 1: 0x6bfd9cad Mirror 11: 0x6bfd9cad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1706c: skipped 000257 1789063103.281131 punch from 0x006c42 to 0x0097ad (0x2b6c bytes) 000259 1789063103.567233 punch from 0x00c69d to 0x00fc82 (0x35e6 bytes) 000260 1789063103.581555 trunc from 0x01706c to 0x0252b6 (0xe24b bytes) 000262 1789063103.691251 write 0x03dc3d thru 0x03ffff (0x23c3 bytes) 000263 1789063103.692553 mapwrite 0x00f2c4 thru 0x0142d0 (0x500d bytes) 000265 1789063104.017073 punch from 0x01b8fd to 0x027148 (0xb84c bytes) 000266 1789063104.045117 mapwrite 0x0210d9 thru 0x025522 (0x444a bytes) 000270 1789063104.272765 write 0x00df28 thru 0x01d8f6 (0xf9cf bytes) 000271 1789063104.277162 trunc from 0x040000 to 0x004993 (0x3b66c bytes) 000276 1789063104.333777 read 0x002b51 thru 0x004993 (0x1e43 bytes) 000277 1789063104.359289 MIRROR_RESYNC 4 mirrors 000280 1789063106.935935 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) 000281 1789063107.467847 mapread 0x000abb thru 0x004993 (0x3ed9 bytes) 000282 1789063107.500333 read 0x002a9a thru 0x004993 (0x1efa bytes) 000284 1789063107.503641 punch from 0x004348 to 0x004993 (0x64c bytes) 000285 1789063107.629085 mapread 0x002147 thru 0x004993 (0x284d bytes) 000287 1789063107.644883 mapwrite 0x03e91d thru 0x03ffff (0x16e3 bytes) 000288 1789063107.744878 mapread 0x02ca0c thru 0x031b2b (0x5120 bytes) 000289 1789063107.762054 trunc from 0x040000 to 0x01ee41 (0x211be bytes) 000290 1789063107.791094 punch from 0x00ecb0 to 0x01a5b2 (0xb903 bytes) 000293 1789063107.801701 mapwrite 0x01866b thru 0x01aca2 (0x2638 bytes) 000297 1789063107.882870 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ee42): Mirror 8: 0x26872122 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ee42): Mirror 8: 0x26872122 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ee42: skipped 000298 1789063108.232385 write 0x03e68b thru 0x03ffff (0x1975 bytes) 000300 1789063108.238737 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x6d86a6d8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x427fe247 Mirror 9: 0x427fe247 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000301 1789063109.467934 mapwrite 0x00a2fc thru 0x0173fc (0xd101 bytes) 000302 1789063109.789345 read 0x00a0e6 thru 0x00a48f (0x3aa bytes) 000304 1789063109.790535 read 0x02241a thru 0x024a93 (0x267a bytes) 000305 1789063109.805116 read 0x010ebc thru 0x01dc2e (0xcd73 bytes) 000306 1789063109.805921 punch from 0x01e87a to 0x02058b (0x1d12 bytes) 000307 1789063109.810727 trunc from 0x040000 to 0x029b1d (0x164e2 bytes) 000309 1789063109.861326 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29b1e): Mirror 9: 0xbd3f6e2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b1e): Mirror 9: 0xbd3f6e2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b1e: skipped 000310 1789063110.191779 write 0x037369 thru 0x03b3fc (0x4094 bytes) 000312 1789063110.193988 mapread 0x00da42 thru 0x015e08 (0x83c7 bytes) 000313 1789063110.211440 mapread 0x03762a thru 0x03b3fc (0x3dd3 bytes) 000316 1789063110.212778 mapread 0x01db27 thru 0x02b733 (0xdc0d bytes) 000317 1789063110.227242 trunc from 0x03b3fd to 0x023d90 (0x1766c bytes) 000318 1789063110.242695 write 0x03326e thru 0x03c1fa (0x8f8d bytes) 000319 1789063110.243987 read 0x00d799 thru 0x013e28 (0x6690 bytes) 000321 1789063110.245781 write 0x003b37 thru 0x012886 (0xed50 bytes) 000322 1789063110.252339 read 0x02e421 thru 0x03ac7c (0xc85c bytes) 000324 1789063110.259413 read 0x0171d2 thru 0x018b97 (0x19c6 bytes) 000326 1789063110.263340 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 9: 0x8242105d (stale) CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 1: 0xdeb2e0d5 Mirror 10: 0xdeb2e0d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1fb: skipped 000329 1789063111.425853 write 0x0256cd thru 0x030bab (0xb4df bytes) 000331 1789063111.549314 punch from 0x00f821 to 0x011bc8 (0x23a8 bytes) 000335 1789063111.587853 punch from 0x039b83 to 0x03c1fa (0x2678 bytes) 000336 1789063111.614829 MIRROR_RESYNC 3 mirrors 000337 1789063112.925748 mapread 0x004424 thru 0x00ccdc (0x88b9 bytes) 000339 1789063112.962333 mapwrite 0x003551 thru 0x012db8 (0xf868 bytes) 000340 1789063113.265306 MIRROR_RESYNC 3 mirrors 000341 1789063115.350297 punch from 0x00c320 to 0x0104b7 (0x4198 bytes) 000342 1789063115.525106 mapread 0x0000eb thru 0x00c62a (0xc540 bytes) 000343 1789063115.575747 read 0x028913 thru 0x0371ba (0xe8a8 bytes) 000344 1789063115.604803 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 9: 0x1584cbf1 (stale) Mirror 10: 0x1584cbf1 (stale) CRC-32 checksum value for chunk [0, 0x3c1fb): Mirror 1: 0x4bbd3802 Mirror 11: 0x4bbd3802 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1fb: skipped 000345 1789063117.776207 punch from 0x01f075 to 0x01fff0 (0xf7c bytes) 000346 1789063117.971475 read 0x01a0f5 thru 0x01bc24 (0x1b30 bytes) 000350 1789063117.989812 read 0x03baf6 thru 0x03c1fa (0x705 bytes) 000351 1789063118.014661 mapread 0x01fcb6 thru 0x02d441 (0xd78c bytes) 000352 1789063118.015758 mapwrite 0x00331f thru 0x00df84 (0xac66 bytes) 000354 1789063118.165361 trunc from 0x03c1fb to 0x00df1a (0x2e2e0 bytes) 000357 1789063118.217072 mapwrite 0x03afee thru 0x03f178 (0x418b bytes) 000358 1789063118.401797 punch from 0x027d36 to 0x02ef30 (0x71fb bytes) 000360 1789063118.409892 mapread 0x00fb09 thru 0x017506 (0x79fe bytes) 000362 1789063118.492109 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f179): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f179): Mirror 9: 0xe7ea11ff (stale) Mirror 10: 0xe7ea11ff (stale) Mirror 11: 0x3c86ed2c (stale) CRC-32 checksum value for chunk [0, 0x3f179): Mirror 1: 0xfd454752 Mirror 12: 0xfd454752 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f179: skipped 000363 1789063120.081440 trunc from 0x03f179 to 0x025610 (0x19b68 bytes) 000364 1789063120.215909 MIRROR_RESYNC 5 mirrors 000365 1789063121.756891 MIRROR_EXTEND to 6 mirrors 000366 1789063123.395371 write 0x0195f7 thru 0x027452 (0xde5c bytes) 000367 1789063123.567883 mapread 0x02187f thru 0x027452 (0x5bd4 bytes) 000369 1789063123.569316 write 0x01624e thru 0x0184ef (0x22a2 bytes) 000370 1789063123.596573 read 0x0037b0 thru 0x00a928 (0x7179 bytes) 000371 1789063123.619266 MIRROR_RESYNC 6 mirrors 000372 1789063125.534618 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) 000373 1789063126.291230 MIRROR_EXTEND to 6 mirrors 000375 1789063127.714685 punch from 0x00812f to 0x0144aa (0xc37c bytes) 000376 1789063127.882672 punch from 0x0013f8 to 0x003797 (0x23a0 bytes) 000378 1789063127.897115 write 0x008251 thru 0x00b213 (0x2fc3 bytes) 000380 1789063127.919157 punch from 0x005c21 to 0x0067e3 (0xbc3 bytes) 000381 1789063127.958742 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27453): Mirror 9: 0x571d68ae (stale) Mirror 11: 0x571d68ae (stale) Mirror 13: 0x571d68ae (stale) Mirror 14: 0x571d68ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27453): Mirror 9: 0x571d68ae (stale) Mirror 11: 0x571d68ae (stale) Mirror 13: 0x571d68ae (stale) Mirror 14: 0x571d68ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27453: skipped 000382 1789063128.473550 write 0x03e3d0 thru 0x03ffff (0x1c30 bytes) 000383 1789063128.474712 punch from 0x003092 to 0x00710b (0x407a bytes) 000384 1789063128.495180 punch from 0x01f6f2 to 0x027c20 (0x852f bytes) 000385 1789063128.502462 mapwrite 0x03fe12 thru 0x03ffff (0x1ee bytes) 000386 1789063128.554484 trunc from 0x040000 to 0x0058d7 (0x3a728 bytes) 000388 1789063128.575951 mapwrite 0x02b884 thru 0x035789 (0x9f06 bytes) 000389 1789063128.695076 mapread 0x01103d thru 0x01a752 (0x9716 bytes) 000390 1789063128.723863 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3578a): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3578a): Mirror 9: 0x3e289a03 (stale) Mirror 11: 0x3e289a03 (stale) Mirror 13: 0x3e289a03 (stale) Mirror 14: 0x3e289a03 (stale) CRC-32 checksum value for chunk [0, 0x3578a): Mirror 1: 0x7dc4104a Mirror 15: 0x7dc4104a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3578a: skipped 000391 1789063130.116306 write 0x001e4c thru 0x006a3a (0x4bef bytes) 000392 1789063130.474536 read 0x02c90f thru 0x035789 (0x8e7b bytes) 000393 1789063130.503120 MIRROR_RESYNC 6 mirrors 000396 1789063133.359654 mapwrite 0x005f8a thru 0x011868 (0xb8df bytes) 000398 1789063133.705404 mapread 0x006a82 thru 0x0098a4 (0x2e23 bytes) 000399 1789063133.706324 punch from 0x01f128 to 0x0215af (0x2488 bytes) 000400 1789063133.733617 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3578a): Mirror 11: 0x8b535c59 (stale) Mirror 13: 0x8b535c59 (stale) Mirror 14: 0x8b535c59 (stale) Mirror 15: 0x8b535c59 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3578a): Mirror 11: 0x8b535c59 (stale) Mirror 13: 0x8b535c59 (stale) Mirror 14: 0x8b535c59 (stale) Mirror 15: 0x8b535c59 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3578a: skipped 000401 1789063134.286732 mapread 0x019ced thru 0x028ec5 (0xf1d9 bytes) 000402 1789063134.330652 trunc from 0x03578a to 0x02c2e1 (0x94a8 bytes) 000403 1789063134.356843 trunc from 0x02c2e2 to 0x019764 (0x12b7d bytes) 000404 1789063134.374064 write 0x03a2af thru 0x03bbe3 (0x1935 bytes) 000408 1789063134.386261 MIRROR_RESYNC 5 mirrors 000410 1789063136.455071 mapread 0x02a480 thru 0x03926e (0xedef bytes) 000411 1789063136.482523 trunc from 0x03bbe4 to 0x024a09 (0x171da bytes) 000415 1789063136.586143 write 0x0294b6 thru 0x02d5e3 (0x412e bytes) 000418 1789063136.591624 read 0x009a06 thru 0x015adc (0xc0d7 bytes) 000419 1789063136.600838 write 0x004a1d thru 0x012714 (0xdcf8 bytes) 000421 1789063136.609585 write 0x02ca6c thru 0x0340e4 (0x7679 bytes) 000423 1789063136.616750 MIRROR_RESYNC 5 mirrors 000424 1789063138.196219 mapwrite 0x02e432 thru 0x031a75 (0x3644 bytes) 000425 1789063138.408558 punch from 0x02b6c3 to 0x02d809 (0x2147 bytes) 000426 1789063138.415451 read 0x013494 thru 0x0135f3 (0x160 bytes) 000427 1789063138.427273 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x340e5): Mirror 13: 0xbee91620 (stale) Mirror 14: 0xbee91620 (stale) Mirror 15: 0xbee91620 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340e5): Mirror 13: 0xbee91620 (stale) Mirror 14: 0xbee91620 (stale) Mirror 15: 0xbee91620 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340e5: skipped 000428 1789063138.953474 read 0x010811 thru 0x012155 (0x1945 bytes) 000430 1789063138.978340 write 0x02c755 thru 0x03b273 (0xeb1f bytes) 000435 1789063139.006538 punch from 0x02a422 to 0x0321d7 (0x7db6 bytes) 000436 1789063139.095730 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b274): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b274): Mirror 13: 0x591e41af (stale) Mirror 14: 0x591e41af (stale) Mirror 15: 0x591e41af (stale) CRC-32 checksum value for chunk [0, 0x3b274): Mirror 1: 0x84fd3fb9 Mirror 16: 0x84fd3fb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b274: skipped 000437 1789063140.696826 trunc from 0x03b274 to 0x03539c (0x5ed7 bytes) 000442 1789063140.831700 read 0x01e5bd thru 0x029fdf (0xba23 bytes) 000443 1789063140.845423 mapwrite 0x02cd51 thru 0x02d0d7 (0x387 bytes) 000444 1789063140.942928 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3539d): Mirror 13: 0x3db6a19c (stale) Mirror 14: 0x3db6a19c (stale) Mirror 15: 0x3db6a19c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3539d): Mirror 13: 0x3db6a19c (stale) Mirror 14: 0x3db6a19c (stale) Mirror 15: 0x3db6a19c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3539d: skipped 000446 1789063141.353208 mapread 0x00a172 thru 0x00f043 (0x4ed2 bytes) 000447 1789063141.372259 write 0x02c771 thru 0x034cb9 (0x8549 bytes) 000448 1789063141.388824 mapread 0x023464 thru 0x02d3d2 (0x9f6f bytes) 000449 1789063141.405434 punch from 0x02256b to 0x030dfa (0xe890 bytes) 000450 1789063141.423855 mapwrite 0x0217ad thru 0x02da42 (0xc296 bytes) 000452 1789063141.513912 trunc from 0x03539d to 0x025c6d (0xf72f bytes) 000453 1789063141.548725 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25c6e): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c6e): Mirror 13: 0xa709276 (stale) Mirror 14: 0xa709276 (stale) Mirror 15: 0xa709276 (stale) CRC-32 checksum value for chunk [0, 0x25c6e): Mirror 1: 0x1b24b334 Mirror 16: 0x1b24b334 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c6e: skipped 000455 1789063143.867803 trunc from 0x025c6e to 0x018ca9 (0xcfc4 bytes) 000460 1789063144.120526 write 0x00cbbd thru 0x01bd2d (0xf171 bytes) 000461 1789063144.130348 mapwrite 0x02ade1 thru 0x03588e (0xaaae bytes) 000463 1789063144.347815 write 0x02d237 thru 0x02ec99 (0x1a63 bytes) 000464 1789063144.348918 mapwrite 0x000183 thru 0x00f552 (0xf3d0 bytes) 000466 1789063144.470321 read 0x018195 thru 0x0202e5 (0x8151 bytes) 000467 1789063144.503419 read 0x0148d4 thru 0x020112 (0xb83f bytes) 000471 1789063144.510892 mapwrite 0x033fb1 thru 0x03ffff (0xc04f bytes) 000472 1789063144.693230 mapread 0x03b38b thru 0x03ffff (0x4c75 bytes) 000473 1789063144.699770 read 0x014677 thru 0x01e874 (0xa1fe bytes) 000475 1789063144.705422 read 0x031e8d thru 0x034e23 (0x2f97 bytes) 000476 1789063144.707207 mapwrite 0x008e31 thru 0x012e7a (0xa04a bytes) 000477 1789063144.797686 mapwrite 0x029f7d thru 0x0304c0 (0x6544 bytes) 000478 1789063144.906028 mapread 0x03f91a thru 0x03ffff (0x6e6 bytes) 000480 1789063144.907753 mapread 0x01b805 thru 0x022290 (0x6a8c bytes) 000482 1789063144.914737 trunc from 0x040000 to 0x026e44 (0x191bb bytes) 000485 1789063144.964419 read 0x022a53 thru 0x026e44 (0x43f2 bytes) 000486 1789063144.967080 mapread 0x00e23d thru 0x01882f (0xa5f3 bytes) 000487 1789063144.975441 mapwrite 0x03dabe thru 0x03ffff (0x2542 bytes) 000489 1789063145.192393 write 0x03a0c4 thru 0x03f976 (0x58b3 bytes) 000490 1789063145.198481 write 0x013fbd thru 0x01d59a (0x95de bytes) 000491 1789063145.199934 write 0x0284cf thru 0x02b215 (0x2d47 bytes) 000492 1789063145.201045 mapread 0x001a26 thru 0x00bbbd (0xa198 bytes) 000493 1789063145.210303 mapread 0x0368f9 thru 0x03c94a (0x6052 bytes) 000495 1789063145.219282 mapread 0x03b91f thru 0x03fa67 (0x4149 bytes) 000497 1789063145.230699 trunc from 0x040000 to 0x029ea6 (0x16159 bytes) 000498 1789063145.285169 punch from 0x00798d to 0x00d1e1 (0x5855 bytes) 000499 1789063145.330341 read 0x02964a thru 0x029ea6 (0x85d bytes) 000504 1789063145.348548 read 0x024f91 thru 0x029ea6 (0x4f16 bytes) 000505 1789063145.355019 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29ea7): Mirror 13: 0xe2c72640 (stale) Mirror 14: 0xe2c72640 (stale) Mirror 15: 0xe2c72640 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ea7): Mirror 13: 0xe2c72640 (stale) Mirror 14: 0xe2c72640 (stale) Mirror 15: 0xe2c72640 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ea7: skipped 000506 1789063145.911714 mapwrite 0x03a2ad thru 0x03ffff (0x5d53 bytes) 000507 1789063146.084859 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xf4354e39 (stale) Mirror 14: 0xf4354e39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xf4354e39 (stale) Mirror 14: 0xf4354e39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000508 1789063146.778227 mapread 0x007e9a thru 0x014d28 (0xce8f bytes) 000510 1789063146.864119 mapwrite 0x00d46d thru 0x01b716 (0xe2aa bytes) 000513 1789063147.129456 read 0x03406b thru 0x03ffff (0xbf95 bytes) 000514 1789063147.199148 read 0x00b404 thru 0x00c91f (0x151c bytes) 000515 1789063147.199693 write 0x006290 thru 0x006fa9 (0xd1a bytes) 000517 1789063147.201089 punch from 0x00c9e1 to 0x0143b2 (0x79d2 bytes) 000518 1789063147.226924 trunc from 0x040000 to 0x013a52 (0x2c5ad bytes) 000519 1789063147.274293 punch from 0x0088ef to 0x01302b (0xa73d bytes) 000522 1789063147.293188 MIRROR_RESYNC 3 mirrors 000523 1789063148.739412 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) 000524 1789063149.257634 mapwrite 0x020a1e thru 0x02ec6f (0xe252 bytes) 000525 1789063149.597379 trunc from 0x02ec70 to 0x0130b3 (0x1bbbc bytes) 000526 1789063149.631408 write 0x011e10 thru 0x0168ae (0x4a9f bytes) 000527 1789063149.653711 read 0x007098 thru 0x00c156 (0x50bf bytes) 000528 1789063149.668213 read 0x01416c thru 0x0168ae (0x2743 bytes) 000529 1789063149.669701 mapread 0x002763 thru 0x00e966 (0xc204 bytes) 000530 1789063149.681552 write 0x0148f5 thru 0x021ac5 (0xd1d1 bytes) 000532 1789063149.685513 write 0x039832 thru 0x03ffff (0x67ce bytes) 000533 1789063149.689305 mapwrite 0x00c5ac thru 0x01174e (0x51a3 bytes) 000535 1789063149.887902 mapread 0x016123 thru 0x017441 (0x131f bytes) 000536 1789063149.888842 read 0x01b2f2 thru 0x01f8aa (0x45b9 bytes) 000539 1789063149.896836 punch from 0x006988 to 0x0135e7 (0xcc60 bytes) 000540 1789063149.906207 mapread 0x000b20 thru 0x00fa52 (0xef33 bytes) 000541 1789063149.955360 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x5c001591 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3bfe72ac Mirror 15: 0x3bfe72ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000543 1789063151.686153 mapread 0x00524e thru 0x01290d (0xd6c0 bytes) 000544 1789063151.740490 write 0x0041a8 thru 0x0115e9 (0xd442 bytes) 000545 1789063151.891388 read 0x006897 thru 0x0148c3 (0xe02d bytes) 000546 1789063151.905585 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x3bfe72ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x3bfe72ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000547 1789063152.343883 write 0x0306c9 thru 0x0350d2 (0x4a0a bytes) 000548 1789063152.354301 read 0x00e4d8 thru 0x0183bc (0x9ee5 bytes) 000549 1789063152.370806 trunc from 0x040000 to 0x029bcd (0x16432 bytes) 000550 1789063152.418258 mapwrite 0x00fb54 thru 0x01793e (0x7deb bytes) 000551 1789063152.526900 write 0x0318bd thru 0x033061 (0x17a5 bytes) 000555 1789063152.535177 read 0x031f4c thru 0x032aae (0xb63 bytes) 000556 1789063152.535925 trunc from 0x033062 to 0x00b104 (0x27f5d bytes) 000558 1789063152.562123 mapwrite 0x019011 thru 0x022844 (0x9834 bytes) 000559 1789063152.803480 read 0x0127f5 thru 0x017b12 (0x531e bytes) 000560 1789063152.804151 write 0x015588 thru 0x01f405 (0x9e7e bytes) 000561 1789063152.805359 read 0x01b99d thru 0x01f1b4 (0x3818 bytes) 000562 1789063152.812329 read 0x0114f4 thru 0x01b9d0 (0xa4dd bytes) 000563 1789063152.814157 read 0x004b54 thru 0x00844a (0x38f7 bytes) 000564 1789063152.815914 read 0x01a913 thru 0x01be80 (0x156e bytes) 000565 1789063152.816580 punch from 0x00fee4 to 0x012e12 (0x2f2f bytes) 000566 1789063152.875193 read 0x01d720 thru 0x020d10 (0x35f1 bytes) 000567 1789063152.875831 mapread 0x00c9a0 thru 0x014d6c (0x83cd bytes) 000568 1789063152.886505 mapread 0x00e7f9 thru 0x014f08 (0x6710 bytes) 000569 1789063152.887329 trunc from 0x022845 to 0x018fba (0x988a bytes) 000570 1789063152.913775 punch from 0x001fff to 0x00ee48 (0xce4a bytes) 000571 1789063152.921553 trunc from 0x018fbb to 0x032f96 (0x19fdc bytes) 000572 1789063152.973221 mapread 0x030332 thru 0x0310bd (0xd8c bytes) 000573 1789063153.011408 mapwrite 0x03b26d thru 0x03ec85 (0x3a19 bytes) 000574 1789063153.218938 MIRROR_RESYNC 2 mirrors 000575 1789063154.308632 MIRROR_RESYNC 2 mirrors 000577 1789063155.098224 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) 000579 1789063155.395447 write 0x03f33f thru 0x03ffff (0xcc1 bytes) 000581 1789063155.434247 mapwrite 0x019476 thru 0x019f02 (0xa8d bytes) 000583 1789063155.634535 punch from 0x002bf3 to 0x006e70 (0x427e bytes) 000584 1789063155.641319 mapwrite 0x014230 thru 0x01825a (0x402b bytes) 000585 1789063155.698263 write 0x00ccf9 thru 0x01b9da (0xece2 bytes) 000591 1789063155.711439 mapread 0x0194a2 thru 0x0217dd (0x833c bytes) 000594 1789063155.723300 write 0x02bf70 thru 0x036142 (0xa1d3 bytes) 000596 1789063155.734591 punch from 0x0198fd to 0x028d30 (0xf434 bytes) 000597 1789063155.756695 write 0x03ea65 thru 0x03ffff (0x159b bytes) 000598 1789063155.763776 mapwrite 0x01de7b thru 0x01ea62 (0xbe8 bytes) 000600 1789063155.870834 write 0x02f759 thru 0x03e30b (0xebb3 bytes) 000601 1789063155.873078 trunc from 0x040000 to 0x00db3d (0x324c2 bytes) 000602 1789063155.927094 mapwrite 0x03e32a thru 0x03ffff (0x1cd6 bytes) 000607 1789063156.184342 trunc from 0x040000 to 0x02d036 (0x12fc9 bytes) 000611 1789063156.217599 mapread 0x007c33 thru 0x00ddfd (0x61cb bytes) 000612 1789063156.232448 read 0x00425a thru 0x00f223 (0xafca bytes) 000613 1789063156.233213 mapwrite 0x027625 thru 0x0305c3 (0x8f9f bytes) 000614 1789063156.385515 read 0x024216 thru 0x02428c (0x77 bytes) 000619 1789063156.396268 read 0x00ab90 thru 0x0189bf (0xde30 bytes) 000620 1789063156.442717 MIRROR_EXTEND to 2 mirrors 000622 1789063158.481800 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) 000624 1789063158.895540 MIRROR_EXTEND to 2 mirrors 000625 1789063161.103565 mapwrite 0x026344 thru 0x0346b5 (0xe372 bytes) 000626 1789063161.670667 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: 0x2532d928 (stale) CRC-32 checksum value for chunk [0, 0x346b6): Mirror 1: 0x648a7eac Mirror 3: 0x648a7eac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x346b6: skipped 000627 1789063163.563409 punch from 0x01aad2 to 0x026ea8 (0xc3d7 bytes) 000628 1789063163.705891 write 0x018115 thru 0x023e8d (0xbd79 bytes) 000629 1789063163.745892 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: 0x2532d928 (stale) Mirror 3: 0x648a7eac (stale) CRC-32 checksum value for chunk [0, 0x346b6): Mirror 1: 0x3bb6fb85 Mirror 4: 0x3bb6fb85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x346b6: skipped 000630 1789063165.947154 MIRROR_RESYNC 4 mirrors 000631 1789063167.116266 read 0x028a53 thru 0x0341a0 (0xb74e bytes) 000632 1789063167.127074 MIRROR_RESYNC 4 mirrors 000634 1789063167.631836 MIRROR_RESYNC 4 mirrors 000635 1789063168.287736 punch from 0x018c79 to 0x02146b (0x87f3 bytes) 000636 1789063168.403758 write 0x018ada thru 0x024c31 (0xc158 bytes) 000637 1789063168.439046 read 0x028ed8 thru 0x02fdbb (0x6ee4 bytes) 000638 1789063168.456886 write 0x006d77 thru 0x010963 (0x9bed bytes) 000639 1789063168.478402 read 0x00a410 thru 0x016145 (0xbd36 bytes) 000640 1789063168.491878 mapwrite 0x02a980 thru 0x02d6d1 (0x2d52 bytes) 000642 1789063168.619126 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x346b6): Mirror 2: 0x3bb6fb85 (stale) Mirror 3: 0x3bb6fb85 (stale) Mirror 4: 0x3bb6fb85 (stale) CRC-32 checksum value for chunk [0, 0x346b6): Mirror 1: 0x9f4ae293 Mirror 5: 0x9f4ae293 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x346b6: skipped 000643 1789063169.726361 punch from 0x00c3aa to 0x00cd55 (0x9ac bytes) 000645 1789063169.905546 trunc from 0x0346b6 to 0x008a3d (0x2bc78 bytes) 000646 1789063169.947004 mapwrite 0x01fb73 thru 0x02e13a (0xe5c8 bytes) 000647 1789063170.096916 mapwrite 0x00e892 thru 0x019aa0 (0xb20f bytes) 000648 1789063170.233886 write 0x03f69f thru 0x03ffff (0x961 bytes) 000650 1789063170.235282 read 0x006e6f thru 0x014431 (0xd5c3 bytes) 000651 1789063170.245051 mapwrite 0x03b7a3 thru 0x03ffff (0x485d bytes) 000652 1789063170.358709 read 0x03b0e4 thru 0x03ffff (0x4f1c bytes) 000656 1789063170.360702 punch from 0x030592 to 0x034035 (0x3aa4 bytes) 000657 1789063170.367904 write 0x0376cd thru 0x03ffff (0x8933 bytes) 000658 1789063170.371985 punch from 0x03248a to 0x033f73 (0x1aea bytes) 000659 1789063170.406440 mapwrite 0x005fe1 thru 0x0099c6 (0x39e6 bytes) 000660 1789063170.515897 mapwrite 0x02f862 thru 0x034016 (0x47b5 bytes) 000661 1789063170.589929 mapread 0x03b3a0 thru 0x03ffff (0x4c60 bytes) 000663 1789063170.593716 punch from 0x0009e5 to 0x00d56d (0xcb89 bytes) 000664 1789063170.604898 mapread 0x00709b thru 0x0128f0 (0xb856 bytes) 000665 1789063170.657812 trunc from 0x040000 to 0x005f6b (0x3a094 bytes) 000666 1789063170.686570 mapwrite 0x01d89a thru 0x028194 (0xa8fb bytes) 000667 1789063170.807348 mapread 0x025d65 thru 0x028194 (0x2430 bytes) 000668 1789063170.808627 mapread 0x020e21 thru 0x02810b (0x72eb bytes) 000669 1789063170.810381 punch from 0x018e9b to 0x027093 (0xe1f9 bytes) 000670 1789063170.815019 read 0x01dee1 thru 0x027122 (0x9242 bytes) 000671 1789063170.825189 trunc from 0x028195 to 0x011c00 (0x16594 bytes) 000672 1789063170.842687 mapread 0x006d03 thru 0x00e16a (0x7468 bytes) 000673 1789063170.844307 read 0x00ea9e thru 0x011c00 (0x3163 bytes) 000675 1789063170.845894 mapwrite 0x03468e thru 0x03ffff (0xb972 bytes) 000676 1789063170.922756 trunc from 0x040000 to 0x02f4db (0x10b24 bytes) 000677 1789063170.940115 punch from 0x008abb to 0x009eef (0x1435 bytes) 000678 1789063170.945285 mapread 0x012324 thru 0x01cf04 (0xabe1 bytes) 000680 1789063170.951226 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2f4dc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f4dc): Mirror 2: 0x9fd03b25 (stale) Mirror 3: 0x9fd03b25 (stale) Mirror 4: 0x9fd03b25 (stale) Mirror 5: 0x21339a35 (stale) CRC-32 checksum value for chunk [0, 0x2f4dc): Mirror 1: 0x2db400ab Mirror 6: 0x2db400ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f4dc: skipped 000681 1789063172.809114 punch from 0x006867 to 0x0069fe (0x198 bytes) 000682 1789063172.946795 write 0x029a24 thru 0x02fe5f (0x643c bytes) 000683 1789063172.963352 trunc from 0x02fe60 to 0x0204e3 (0xf97c bytes) 000684 1789063172.987178 trunc from 0x0204e4 to 0x0207c3 (0x2e0 bytes) 000687 1789063173.005249 punch from 0x017d3e to 0x01adfb (0x30be bytes) 000688 1789063173.015806 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x207c4): Mirror 2: 0x98b313c7 (stale) Mirror 4: 0x98b313c7 (stale) Mirror 5: 0x910a5b4a (stale) Mirror 6: 0xc988675c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x207c4): Mirror 2: 0x98b313c7 (stale) Mirror 4: 0x98b313c7 (stale) Mirror 5: 0x910a5b4a (stale) Mirror 6: 0xc988675c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x207c4: skipped 000689 1789063173.449844 write 0x0085a8 thru 0x0135a3 (0xaffc bytes) 000691 1789063173.470859 write 0x022436 thru 0x02c4c5 (0xa090 bytes) 000692 1789063173.473700 write 0x030eee thru 0x038f75 (0x8088 bytes) 000693 1789063173.474686 mapwrite 0x0152b8 thru 0x0251e8 (0xff31 bytes) 000694 1789063173.591842 write 0x0269f9 thru 0x02895e (0x1f66 bytes) 000695 1789063173.601311 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f76): Mirror 2: 0xdf2bf55e (stale) Mirror 4: 0xdf2bf55e (stale) Mirror 5: 0xdd952109 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f76): Mirror 2: 0xdf2bf55e (stale) Mirror 4: 0xdf2bf55e (stale) Mirror 5: 0xdd952109 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f76: skipped 000697 1789063174.174348 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38f76): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f76): Mirror 2: 0xdf2bf55e (stale) Mirror 4: 0xdf2bf55e (stale) Mirror 5: 0xdd952109 (stale) CRC-32 checksum value for chunk [0, 0x38f76): Mirror 1: 0x682c109 Mirror 6: 0x682c109 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f76: skipped 000699 1789063175.002273 mapwrite 0x029c8c thru 0x0329ee (0x8d63 bytes) 000700 1789063175.482113 punch from 0x036a80 to 0x038f75 (0x24f6 bytes) 000701 1789063175.516505 trunc from 0x038f76 to 0x0012f7 (0x37c7e bytes) 000702 1789063175.680271 mapread 0x0010ee thru 0x0012f7 (0x20a bytes) 000703 1789063175.707910 punch from 0x00064d to 0x0012f7 (0xcab bytes) 000705 1789063175.720746 mapwrite 0x02d0b0 thru 0x035409 (0x835a bytes) 000706 1789063175.841956 read 0x018db8 thru 0x019d7a (0xfc3 bytes) 000707 1789063175.858559 trunc from 0x03540a to 0x021d81 (0x13688 bytes) 000710 1789063175.898634 write 0x01fad0 thru 0x02f27a (0xf7ab bytes) 000711 1789063175.922519 MIRROR_RESYNC 5 mirrors 000712 1789063177.793329 write 0x029729 thru 0x0381d3 (0xeaab bytes) 000717 1789063177.988735 write 0x029922 thru 0x02dda2 (0x4481 bytes) 000718 1789063177.989895 mapwrite 0x02e427 thru 0x03a455 (0xc02f bytes) 000719 1789063178.187535 punch from 0x0004ee to 0x0041cc (0x3cdf bytes) 000720 1789063178.204001 mapread 0x023035 thru 0x02ff0e (0xceda bytes) 000721 1789063178.220343 punch from 0x025579 to 0x026c9b (0x1723 bytes) 000724 1789063178.241223 trunc from 0x03a456 to 0x000f12 (0x39543 bytes) 000729 1789063178.360036 read 0x00018e thru 0x000f12 (0xd85 bytes) 000730 1789063178.380687 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf13): Mirror 2: 0x7484fad1 (stale) Mirror 5: 0x7484fad1 (stale) Mirror 6: 0x7484fad1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf13): Mirror 2: 0x7484fad1 (stale) Mirror 5: 0x7484fad1 (stale) Mirror 6: 0x7484fad1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf13: skipped 000731 1789063179.017796 mapwrite 0x03c9ae thru 0x03e1d7 (0x182a bytes) 000733 1789063179.297544 trunc from 0x03e1d8 to 0x035b69 (0x866e bytes) 000734 1789063179.346005 mapread 0x013289 thru 0x01446d (0x11e5 bytes) 000736 1789063179.367244 mapread 0x01dc90 thru 0x02d116 (0xf487 bytes) 000739 1789063179.399947 punch from 0x020419 to 0x02ac12 (0xa7fa bytes) 000740 1789063179.405420 mapread 0x000d0f thru 0x00eb0f (0xde01 bytes) 000741 1789063179.422107 mapwrite 0x027395 thru 0x02e95f (0x75cb bytes) 000742 1789063179.507274 write 0x03e0b1 thru 0x03ffff (0x1f4f bytes) 000743 1789063179.513036 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xceab71c3 (stale) Mirror 6: 0xceab71c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xceab71c3 (stale) Mirror 6: 0xceab71c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000746 1789063179.979619 mapwrite 0x039fae thru 0x03ffff (0x6052 bytes) 000747 1789063180.069117 read 0x01f6e7 thru 0x0292f7 (0x9c11 bytes) 000748 1789063180.080829 mapwrite 0x022464 thru 0x024a12 (0x25af bytes) 000749 1789063180.145853 trunc from 0x040000 to 0x00c080 (0x33f7f bytes) 000750 1789063180.174950 mapread 0x003369 thru 0x009e15 (0x6aad bytes) 000751 1789063180.190362 read 0x0059d6 thru 0x00c080 (0x66ab bytes) 000752 1789063180.191363 mapwrite 0x0345cb thru 0x0385fb (0x4031 bytes) 000753 1789063180.326792 punch from 0x035dc1 to 0x0385fb (0x283b bytes) 000754 1789063180.335501 punch from 0x028e8d to 0x036d82 (0xdef6 bytes) 000755 1789063180.347787 read 0x00c78d thru 0x016ca7 (0xa51b bytes) 000756 1789063180.350554 trunc from 0x0385fc to 0x01876d (0x1fe8e bytes) 000757 1789063180.384861 mapread 0x0137ef thru 0x01876d (0x4f7f bytes) 000758 1789063180.385802 write 0x03a792 thru 0x03ffff (0x586e bytes) 000759 1789063180.400502 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xceab71c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xceab71c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000760 1789063180.907781 write 0x010aee thru 0x018e92 (0x83a5 bytes) 000761 1789063180.931829 read 0x01ec7d thru 0x0219e2 (0x2d66 bytes) 000762 1789063180.946462 read 0x0192b9 thru 0x026a22 (0xd76a bytes) 000763 1789063180.980137 mapwrite 0x0145e6 thru 0x016c2e (0x2649 bytes) 000764 1789063181.047296 punch from 0x01a753 to 0x01d5ef (0x2e9d bytes) 000765 1789063181.052700 mapread 0x0174d8 thru 0x02475d (0xd286 bytes) 000766 1789063181.077018 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xceab71c3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3ee0ae1d Mirror 7: 0x3ee0ae1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000767 1789063182.693834 read 0x03ae02 thru 0x03ffff (0x51fe bytes) 000768 1789063182.740536 read 0x03b7a4 thru 0x03ffff (0x485c bytes) 000769 1789063182.751957 punch from 0x0191ba to 0x01aff2 (0x1e39 bytes) 000770 1789063182.880736 MIRROR_RESYNC 3 mirrors 000771 1789063184.433758 punch from 0x029671 to 0x035590 (0xbf20 bytes) 000772 1789063184.511079 read 0x02117d thru 0x02dcaf (0xcb33 bytes) 000773 1789063184.528320 mapread 0x01f424 thru 0x02af8a (0xbb67 bytes) 000774 1789063184.542289 mapread 0x02617e thru 0x02c415 (0x6298 bytes) 000776 1789063184.545685 punch from 0x0082ae to 0x00c42a (0x417d bytes) 000777 1789063184.559208 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x3ee0ae1d (stale) Mirror 7: 0x3ee0ae1d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3ee0ae1d Mirror 8: 0x3ee0ae1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x3ff20 000778 1789063186.256741 trunc from 0x040000 to 0x03ff1f (0xe0 bytes) 000779 1789063186.401428 read 0x0051ca thru 0x008a94 (0x38cb bytes) 000780 1789063186.409037 punch from 0x00b2b1 to 0x01a6ec (0xf43c bytes) 000781 1789063186.415453 mapwrite 0x008d39 thru 0x00dbdd (0x4ea5 bytes) 000782 1789063186.505957 write 0x023cd4 thru 0x02724d (0x357a bytes) 000783 1789063186.541283 MIRROR_RESYNC 4 mirrors 000784 1789063187.920854 MIRROR_EXTEND to 5 mirrors 000785 1789063189.017327 mapwrite 0x0083f8 thru 0x00d637 (0x5240 bytes) 000786 1789063189.212486 mapwrite 0x03a881 thru 0x03c188 (0x1908 bytes) 000788 1789063189.284674 punch from 0x0100da to 0x01416a (0x4091 bytes) 000790 1789063189.292519 mapwrite 0x0369ab thru 0x03ffff (0x9655 bytes) 000791 1789063189.389777 write 0x015190 thru 0x020602 (0xb473 bytes) 000792 1789063189.405141 mapwrite 0x03b1f8 thru 0x03ffff (0x4e08 bytes) 000793 1789063189.462937 punch from 0x01baff to 0x0275b3 (0xbab5 bytes) 000794 1789063189.469896 mapwrite 0x00786f thru 0x0097a7 (0x1f39 bytes) 000798 1789063189.525650 trunc from 0x040000 to 0x03a0eb (0x5f14 bytes) 000799 1789063189.548330 write 0x03983b thru 0x03ffff (0x67c5 bytes) 000801 1789063189.551931 trunc from 0x040000 to 0x00233b (0x3dcc4 bytes) 000802 1789063189.577662 trunc from 0x00233c to 0x00a431 (0x80f6 bytes) 000804 1789063189.600473 write 0x025f41 thru 0x02f2ff (0x93bf bytes) 000806 1789063189.603956 mapwrite 0x039d17 thru 0x03ffff (0x62e9 bytes) 000809 1789063189.680414 trunc from 0x040000 to 0x0032b3 (0x3cd4c bytes) 000810 1789063189.704764 write 0x007b41 thru 0x01687d (0xed3d bytes) 000811 1789063189.708086 read 0x0015ce thru 0x00e00d (0xca40 bytes) 000812 1789063189.716078 trunc from 0x01687e to 0x039520 (0x22ca3 bytes) 000813 1789063189.740566 punch from 0x027b36 to 0x03072a (0x8bf5 bytes) 000814 1789063189.754870 read 0x007d3c thru 0x01683f (0xeb04 bytes) 000816 1789063189.756046 punch from 0x02ab8e to 0x02be9f (0x1312 bytes) 000817 1789063189.764619 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39521): Mirror 7: 0x56343511 (stale) Mirror 8: 0x56343511 (stale) Mirror 9: 0x56343511 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39521): Mirror 7: 0x56343511 (stale) Mirror 8: 0x56343511 (stale) Mirror 9: 0x56343511 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39521: skipped 000818 1789063190.196553 mapwrite 0x02dc67 thru 0x03588a (0x7c24 bytes) 000820 1789063190.274421 mapwrite 0x0368c7 thru 0x037b1d (0x1257 bytes) 000822 1789063190.316271 mapwrite 0x02affb thru 0x0354f1 (0xa4f7 bytes) 000824 1789063190.373590 trunc from 0x039521 to 0x0243f5 (0x1512b bytes) 000825 1789063190.393463 read 0x02021e thru 0x0243f5 (0x41d8 bytes) 000828 1789063190.406112 mapwrite 0x01ec6d thru 0x02a586 (0xb91a bytes) 000829 1789063190.505913 mapwrite 0x0253fc thru 0x032235 (0xce3a bytes) 000831 1789063190.601034 mapwrite 0x019a13 thru 0x029910 (0xfefe bytes) 000834 1789063190.657612 read 0x01579f thru 0x019cf4 (0x4556 bytes) 000836 1789063190.679745 read 0x008fb1 thru 0x0164cf (0xd51f bytes) 000839 1789063190.681377 write 0x009f15 thru 0x015950 (0xba3c bytes) 000840 1789063190.684478 write 0x008d1d thru 0x016ff6 (0xe2da bytes) 000842 1789063190.687365 mapread 0x02d4cd thru 0x02ef43 (0x1a77 bytes) 000843 1789063190.689091 punch from 0x025d59 to 0x0286d7 (0x297f bytes) 000845 1789063190.710120 read 0x01514a thru 0x017074 (0x1f2b bytes) 000848 1789063190.712596 read 0x005857 thru 0x00e434 (0x8bde bytes) 000850 1789063190.713865 mapwrite 0x01d263 thru 0x026399 (0x9137 bytes) 000853 1789063190.786668 punch from 0x01fba7 to 0x024419 (0x4873 bytes) 000854 1789063190.793347 mapwrite 0x022f86 thru 0x02e258 (0xb2d3 bytes) 000858 1789063190.886644 read 0x004c1e thru 0x005750 (0xb33 bytes) 000861 1789063190.888468 mapread 0x00e915 thru 0x012903 (0x3fef bytes) 000863 1789063190.892494 write 0x0335bc thru 0x0372ef (0x3d34 bytes) 000865 1789063190.895396 write 0x02add9 thru 0x037555 (0xc77d bytes) 000868 1789063190.903666 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37556): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37556): Mirror 7: 0xd5c289b8 (stale) Mirror 8: 0xd5c289b8 (stale) Mirror 9: 0xd5c289b8 (stale) CRC-32 checksum value for chunk [0, 0x37556): Mirror 1: 0x8676ea41 Mirror 10: 0x8676ea41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37556: skipped 000869 1789063192.596717 read 0x03641e thru 0x037555 (0x1138 bytes) 000871 1789063192.657830 mapread 0x026a89 thru 0x02ae92 (0x440a bytes) 000873 1789063192.758831 punch from 0x0019cf to 0x004d66 (0x3398 bytes) 000874 1789063192.977659 mapread 0x02240f thru 0x02707b (0x4c6d bytes) 000876 1789063192.991352 punch from 0x02330a to 0x02c699 (0x9390 bytes) 000879 1789063193.034138 mapwrite 0x012280 thru 0x01a58d (0x830e bytes) 000880 1789063193.163861 mapwrite 0x00a8da thru 0x014761 (0x9e88 bytes) 000882 1789063193.246876 punch from 0x01df1a to 0x026ce7 (0x8dce bytes) 000883 1789063193.257372 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37556): Mirror 8: 0xd5c289b8 (stale) Mirror 9: 0xd5c289b8 (stale) Mirror 10: 0x8676ea41 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37556): Mirror 8: 0xd5c289b8 (stale) Mirror 9: 0xd5c289b8 (stale) Mirror 10: 0x8676ea41 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37556: skipped 000884 1789063193.732024 punch from 0x01cf72 to 0x022f85 (0x6014 bytes) 000885 1789063193.748844 mapwrite 0x03e62c thru 0x03ffff (0x19d4 bytes) 000886 1789063193.886179 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x1f03e720 (stale) Mirror 9: 0x1f03e720 (stale) Mirror 10: 0xe61d9fee (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x596cf7a0 Mirror 11: 0x596cf7a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000887 1789063195.100942 mapread 0x0192b7 thru 0x022cd2 (0x9a1c bytes) 000888 1789063195.161096 read 0x0100d1 thru 0x01f05d (0xef8d bytes) 000889 1789063195.212784 mapread 0x03ec00 thru 0x03ffff (0x1400 bytes) 000890 1789063195.247692 mapwrite 0x013f4c thru 0x022f6f (0xf024 bytes) 000891 1789063195.459168 mapread 0x039be9 thru 0x03ffff (0x6417 bytes) 000892 1789063195.476636 trunc from 0x040000 to 0x001c0e (0x3e3f1 bytes) 000893 1789063195.509307 mapread 0x00030e thru 0x001c0e (0x1901 bytes) 000896 1789063195.523831 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c0f): Mirror 8: 0x29371e3b (stale) Mirror 10: 0x29371e3b (stale) Mirror 11: 0x29371e3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c0f): Mirror 8: 0x29371e3b (stale) Mirror 10: 0x29371e3b (stale) Mirror 11: 0x29371e3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c0f: skipped 000897 1789063195.878010 read 0x0011f3 thru 0x001c0e (0xa1c bytes) 000899 1789063195.885958 mapread 0x0017ea thru 0x001c0e (0x425 bytes) 000903 1789063195.892110 mapwrite 0x00a7b7 thru 0x01233e (0x7b88 bytes) 000905 1789063196.029417 MIRROR_RESYNC 4 mirrors 000907 1789063197.282792 read 0x006f4e thru 0x0099a5 (0x2a58 bytes) 000910 1789063197.294541 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) 000911 1789063197.633371 mapwrite 0x0240b7 thru 0x027198 (0x30e2 bytes) 000912 1789063197.926899 mapread 0x000879 thru 0x01024c (0xf9d4 bytes) 000913 1789063197.966924 MIRROR_RESYNC 3 mirrors 000914 1789063199.663272 MIRROR_RESYNC 3 mirrors 000920 1789063200.518522 read 0x0215c8 thru 0x027198 (0x5bd1 bytes) 000922 1789063200.529513 write 0x01f0fe thru 0x020a61 (0x1964 bytes) 000923 1789063200.703938 read 0x003338 thru 0x010eaa (0xdb73 bytes) 000924 1789063200.726178 write 0x004464 thru 0x0108d1 (0xc46e bytes) 000928 1789063200.732261 read 0x012bce thru 0x01d0b9 (0xa4ec bytes) 000929 1789063200.739240 MIRROR_RESYNC 3 mirrors 000930 1789063202.513062 trunc from 0x027199 to 0x022ed8 (0x42c0 bytes) 000931 1789063202.714279 trunc from 0x022ed9 to 0x03106a (0xe192 bytes) 000932 1789063202.785404 read 0x021759 thru 0x023488 (0x1d30 bytes) 000933 1789063202.795407 mapwrite 0x0331ca thru 0x03e747 (0xb57e bytes) 000934 1789063202.925321 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e748): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e748): Mirror 10: 0xf86c05b (stale) Mirror 11: 0xf86c05b (stale) CRC-32 checksum value for chunk [0, 0x3e748): Mirror 1: 0x4bdbcb57 Mirror 12: 0x4bdbcb57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e748: skipped 000935 1789063204.430234 punch from 0x020350 to 0x022446 (0x20f7 bytes) 000936 1789063204.564533 write 0x00ea36 thru 0x01d886 (0xee51 bytes) 000938 1789063204.577179 trunc from 0x03e748 to 0x03d205 (0x1542 bytes) 000939 1789063204.616395 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d206): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d206): Mirror 10: 0x9d9f71c1 (stale) Mirror 11: 0x9d9f71c1 (stale) Mirror 12: 0xb3536f9e (stale) CRC-32 checksum value for chunk [0, 0x3d206): Mirror 1: 0xa2a72829 Mirror 13: 0xa2a72829 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d206: skipped 000940 1789063205.937183 MIRROR_RESYNC 5 mirrors 000941 1789063207.069717 MIRROR_EXTEND to 6 mirrors 000942 1789063208.185708 write 0x02ddec thru 0x03c789 (0xe99e bytes) 000944 1789063208.275593 trunc from 0x03d206 to 0x0154f2 (0x27d13 bytes) 000945 1789063208.297528 write 0x018884 thru 0x025724 (0xcea1 bytes) 000947 1789063208.299801 punch from 0x01840a to 0x0220c0 (0x9cb7 bytes) 000949 1789063208.324066 trunc from 0x025725 to 0x0366d8 (0x10fb4 bytes) 000950 1789063208.350157 mapwrite 0x0249ba thru 0x028250 (0x3897 bytes) 000951 1789063208.439323 mapwrite 0x028e9e thru 0x02a8f2 (0x1a55 bytes) 000952 1789063208.496289 trunc from 0x0366d9 to 0x02ab36 (0xbba2 bytes) 000953 1789063208.539759 mapread 0x01ffb3 thru 0x0260af (0x60fd bytes) 000954 1789063208.557348 punch from 0x0099df to 0x014226 (0xa848 bytes) 000955 1789063208.564110 punch from 0x00c70f to 0x014801 (0x80f3 bytes) 000956 1789063208.587184 trunc from 0x02ab37 to 0x034e67 (0xa331 bytes) 000958 1789063208.662307 trunc from 0x034e68 to 0x03f248 (0xa3e1 bytes) 000959 1789063208.699291 mapwrite 0x00ed6e thru 0x0156e1 (0x6974 bytes) 000961 1789063208.836761 trunc from 0x03f249 to 0x022343 (0x1cf05 bytes) 000963 1789063208.868541 mapread 0x002f8b thru 0x00f0ba (0xc130 bytes) 000965 1789063208.883745 mapread 0x00f4ad thru 0x013349 (0x3e9d bytes) 000969 1789063208.896818 MIRROR_RESYNC 6 mirrors 000970 1789063210.072851 mapwrite 0x022185 thru 0x0291a7 (0x7023 bytes) 000972 1789063210.232143 write 0x00a57d thru 0x0199fe (0xf482 bytes) 000973 1789063210.246099 mapwrite 0x00d1f5 thru 0x017013 (0x9e1f bytes) 000974 1789063210.295254 trunc from 0x0291a8 to 0x0144d4 (0x14cd3 bytes) 000975 1789063210.321649 read 0x007cf7 thru 0x009097 (0x13a1 bytes) 000977 1789063210.334616 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x144d5): Mirror 11: 0x6ad3e785 (stale) Mirror 12: 0x6ad3e785 (stale) Mirror 13: 0x6ad3e785 (stale) Mirror 14: 0x6ad3e785 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x144d5): Mirror 11: 0x6ad3e785 (stale) Mirror 12: 0x6ad3e785 (stale) Mirror 13: 0x6ad3e785 (stale) Mirror 14: 0x6ad3e785 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x144d5: skipped 000979 1789063210.834652 mapwrite 0x02c757 thru 0x036fc5 (0xa86f bytes) 000980 1789063210.943952 write 0x0356c9 thru 0x03c3ce (0x6d06 bytes) 000981 1789063210.945290 punch from 0x00dabf to 0x017b43 (0xa085 bytes) 000982 1789063210.963084 read 0x014c06 thru 0x016474 (0x186f bytes) 000983 1789063210.971294 mapwrite 0x03c931 thru 0x03ffff (0x36cf bytes) 000984 1789063211.177176 write 0x02eef7 thru 0x03eb64 (0xfc6e bytes) 000985 1789063211.178773 write 0x035608 thru 0x03ffff (0xa9f8 bytes) 000986 1789063211.189264 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x7bf9f7be (stale) Mirror 12: 0x7bf9f7be (stale) Mirror 13: 0x7bf9f7be (stale) Mirror 14: 0x7bf9f7be (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xedb7ef83 Mirror 15: 0xedb7ef83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1789063212.466418 mapread 0x010432 thru 0x012ac9 (0x2698 bytes) 000988 1789063212.508946 write 0x029558 thru 0x02ae2e (0x18d7 bytes) 000989 1789063212.627727 read 0x02a36a thru 0x02c5db (0x2272 bytes) 000990 1789063212.629084 mapread 0x025b86 thru 0x028e4d (0x32c8 bytes) 000991 1789063212.640555 read 0x02ef70 thru 0x03be21 (0xceb2 bytes) 000992 1789063212.658212 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x16d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x7bf9f7be (stale) Mirror 12: 0x7bf9f7be (stale) Mirror 13: 0x7bf9f7be (stale) Mirror 14: 0x7bf9f7be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x7bf9f7be (stale) Mirror 12: 0x7bf9f7be (stale) Mirror 13: 0x7bf9f7be (stale) Mirror 14: 0x7bf9f7be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000993 1789063213.281901 write 0x00ffcb thru 0x010480 (0x4b6 bytes) 000995 1789063213.327465 trunc from 0x040000 to 0x02f14d (0x10eb2 bytes) 000999 1789063213.392564 trunc from 0x02f14e to 0x029c49 (0x5504 bytes) 001000 1789063213.435847 mapwrite 0x0090e0 thru 0x014d83 (0xbca4 bytes) All operations completed A-OK! PASS 70a (179s) == sanity-flr test 71: check mirror extend parent fid ==== 14:00:22 (1789063222) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 14:00:28 (1789063228) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242946 s, 8.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162614 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (15s) == sanity-flr test 200a: stress test ===================== 14:00:42 (1789063242) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6523200 .. Extending file size to 7754080 .. Extending file size to 9235328 .. Extending file size to 8429408 .. Extending file size to 9144160 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 57399 57400 57401 57404 57405 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) 192.168.201.118@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (97s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:02:19 (1789063339) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6817664 .. Extending file size to 4082560 .. Extending file size to 8071904 .. Extending file size to 9099360 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6850496 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7194944 .. Extending file size to 7798880 .. Extending file size to 8570080 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8626144 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58730 58731 58733 58734 58737 done 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) 192.168.201.118@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (91s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:03:50 (1789063430) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161693 s, 13.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.248803 s, 8.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x30df91d9 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:04:16 (1789063456) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.319793 s, 6.6 MB/s PASS 202 (13s) == sanity-flr test 203: mirror file preserve mirror ID === 14:04:29 (1789063469) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2a4:0x0] } PASS 203 (11s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:04:40 (1789063480) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.280259 s, 7.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:04:51 (1789063491) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128689 s, 16.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:05:01 (1789063501) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.323764 s, 6.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:05:39 (1789063539) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg118-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199914 s, 5.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:06:44 (1789063604) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg118-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.355273 s, 5.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (71s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:07:55 (1789063675) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg118-server mds-ost sync done. Creating new pool test_204f oleg118-server: Pool lustre.test_204f created Adding targets to pool oleg118-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg118-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.308145 s, 6.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg118-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg118-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg118-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:09:19 (1789063759) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:09:28 (1789063768) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:09:38 (1789063778) oleg118-server: Pool lustre.test_206 created Adding targets to pool oleg118-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg118-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg118-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg118-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg118-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:10:07 (1789063807) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630286 s, 16.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582439 s, 18.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xaf0c3b2d Mirror 3: 0xaf0c3b2d PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:10:19 (1789063819) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:10:24 (1789063824) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:10:29 (1789063829) Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.210405 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x9f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x15a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x15a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x15a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe50000 Read this (4 bytes): BBBB 192.168.201.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg118-client.virtnet /mnt/lustre2 (opts:) PASS 209a (48s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:11:17 (1789063877) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0735746 s, 111 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:11:26 (1789063886) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055528 s, 18.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e9 lmm_fid: [0x200000404:0x1e9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e9 lmm_fid: [0x200000404:0x1e9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834409 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 14:11:28.000000000 -0400 Modify: 2026-09-10 14:11:28.000000000 -0400 Change: 2026-09-10 14:11:30.000000000 -0400 Birth: 2026-09-10 14:11:28.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:11:37 (1789063897) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:11:41 (1789063901) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.513764 s, 79.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0432165 s, 94.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 10 14:11 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1eb:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 10 14:11 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:11:51 (1789063911) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:12:02 (1789063922) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 2202 sec =========== 14:12:13 (1789063933) === sanity-flr: start cleanup 14:12:15 (1789063935) === === sanity-flr: finish cleanup 14:12:22 (1789063942) ===