-----============= acceptance-small: sanity-flr ============----- Fri Aug 7 07:36:36 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg637-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 07:36:55 (1786102615) === oleg637-client.virtnet: executing check_config_client /mnt/lustre oleg637-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg637-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff934d902ea800.idle_timeout=debug osc.lustre-OST0001-osc-ffff934d902ea800.idle_timeout=debug disable quota as required oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 07:37:29 (1786102649) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7650] == sanity-flr test 0a: lfs mirror create with -N option == 07:37:31 (1786102651) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:37:46 (1786102666) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:37:49 (1786102669) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:37:52 (1786102672) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:38:17 (1786102697) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:38:20 (1786102700) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 07:38:23 (1786102703) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff934d902ea800.stats=clear osc.lustre-OST0001-osc-ffff934d902ea800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:38:30 (1786102710) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:38:40 (1786102720) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:38:47 (1786102727) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:38:50 (1786102730) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:38:58 (1786102738) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 07:39:06 (1786102746) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:39:12 (1786102752) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:39:15 (1786102755) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:39:23 (1786102763) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:39:32 (1786102772) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:39:37 (1786102777) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.19757 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900964 s, 11.6 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:39:47 (1786102787) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.550355 s, 34.3 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:39:54 (1786102794) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:40:04 (1786102804) Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff934d902ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff934d902ea800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff934d902ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff934d902ea800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg637-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (69s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:41:13 (1786102873) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.402649 s, 26.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff934d902ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff934d902ea800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:41:59 (1786102919) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff934d902ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff934d902ea800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff934d902ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff934d902ea800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg637-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (70s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:43:10 (1786102990) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.19642 s, 12.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 7 07:43 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.907s user 0m0.001s sys 0m0.159s Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff934d902ea800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff934d902ea800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.883s user 0m0.001s sys 0m0.170s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff934d902ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff934d902ea800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.192s user 0m0.000s sys 0m0.156s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg637-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg637-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff934d902ea800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff934d902ea800.ost_server_uuid in FULL state after 0 sec PASS 33b (83s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:44:33 (1786103073) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:44:37 (1786103077) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:44:40 (1786103080) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 07:44:43 (1786103083) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 07:44:50 (1786103090) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.401333 s, 31.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:45:02 (1786103102) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294936 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236882 s, 44.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:45:09 (1786103109) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:45:17 (1786103117) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00275987 s, 371 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00147253 s, 695 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105543 s, 970 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00204294 s, 501 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 07:45:26 (1786103126) create test files with size 15739676 .. a264b60bed1780dd93b5eb097cca2c9e - 354bddbb5005ffdf8eb5728932a3e8c4 - 4331a31cc733af03e9fa464cc0d07c0f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg637-client.virtnet /mnt/lustre (opts:) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 07:45:48 (1786103148) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:46:01 (1786103161) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:46:07 (1786103167) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807103 s, 13.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365185 s, 28.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 07:46:16 (1786103176) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115133 s, 18.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.34732 s, 12.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.342505 s, 6.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 07:46:32 (1786103192) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 07:46:35 (1786103195) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:46:38 (1786103198) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567422 s, 18.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 07:46:46 (1786103206) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:46:49 (1786103209) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:46:59 (1786103219) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.263679 s, 39.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg637-client.virtnet /mnt/lustre (opts:) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:47:09 (1786103229) llite.lustre-ffff934d903f2000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.333091 s, 31.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg637-client.virtnet /mnt/lustre (opts:) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff934d85ead000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:47:23 (1786103243) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:47:58 (1786103278) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:48:32 (1786103312) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 07:48:35 (1786103315) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.280465 s, 11.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.182206 s, 17.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4b2fdaf6 (nosync,stale) Mirror 2: 0x4b2fdaf6 (nosync,stale) ** verify mirror 2 doesn't change original checksum: e1e671c1c8914317af69af83588836d7 - mirror 1 checksum: 7e1f8cfd28deb56fce1d90a0933d79f0 - mirror 2 checksum: e1e671c1c8914317af69af83588836d7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-07T07:48:38-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-07T07:48:38-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e1e671c1c8914317af69af83588836d7 - mirror 1 checksum: 7e1f8cfd28deb56fce1d90a0933d79f0 - mirror 2 checksum: 7e1f8cfd28deb56fce1d90a0933d79f0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-07T07:48:38-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-07T07:48:38-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:48:46 (1786103326) Failing mds1 on oleg637-server Stopping /mnt/lustre-mds1 (opts:) on oleg637-server 07:48:53 (1786103333) shut down facet: mds1 facet_host: oleg637-server facet_failover_host: oleg637-server Failover mds1 to oleg637-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-MDT0000 07:49:17 (1786103357) targets are mounted 07:49:18 (1786103358) facet_failover done oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189977 s, 55.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:49:34 (1786103374) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:49:37 (1786103377) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:49:40 (1786103380) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00618871 s, 2.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:49:48 (1786103388) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:49:52 (1786103392) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0160017 s, 256 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0541416 s, 75.7 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:49:59 (1786103399) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:50:21 (1786103421) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:50:49 (1786103449) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:51:12 (1786103472) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:51:35 (1786103495) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 07:51:41 (1786103501) mirror create pid 54320 mirror split pid 54321 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54320 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54321 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 07:52:48 (1786103568) Seed set to 3570 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x97a4 000001 1786103570.327384 trunc from 00000000 to 0x0097a3 (0x97a4 bytes) 000002 1786103570.358778 write 0x01ae6a thru 0x024cce (0x9e65 bytes) 000004 1786103570.365525 mapread 0x01f671 thru 0x01f801 (0x191 bytes) 000006 1786103570.367217 mapread 0x00ec13 thru 0x016ba1 (0x7f8f bytes) 000007 1786103570.406300 read 0x01de5a thru 0x024cce (0x6e75 bytes) 000008 1786103570.406977 read 0x01bb7b thru 0x024cce (0x9154 bytes) 000009 1786103570.407618 mapwrite 0x00817a thru 0x011263 (0x90ea bytes) 000010 1786103570.551814 mapwrite 0x038717 thru 0x03f5b3 (0x6e9d bytes) 000011 1786103570.670690 read 0x02d7a5 thru 0x036fbb (0x9817 bytes) 000012 1786103570.675491 read 0x0088b1 thru 0x016f0a (0xe65a bytes) 000016 1786103570.689719 mapread 0x010cd1 thru 0x01a8da (0x9c0a bytes) 000017 1786103570.702662 read 0x00e9a6 thru 0x0158ab (0x6f06 bytes) 000018 1786103570.706853 mapwrite 0x03621a thru 0x03db5e (0x7945 bytes) truncating to largest ever: 0x28f1e 000019 1786103570.798728 trunc from 0x03f5b4 to 0x028f1d (0x16696 bytes) 000020 1786103570.824780 write 0x039ea7 thru 0x03ffff (0x6159 bytes) 000021 1786103570.835638 read 0x018c6e thru 0x021e56 (0x91e9 bytes) 000022 1786103570.839150 mapread 0x02a1cf thru 0x02eebf (0x4cf1 bytes) 000023 1786103570.852213 write 0x03238e thru 0x03bba7 (0x981a bytes) 000024 1786103570.853922 read 0x00a590 thru 0x017699 (0xd10a bytes) 000027 1786103570.857192 mapread 0x025b88 thru 0x032cd6 (0xd14f bytes) 000035 1786103570.864807 mapread 0x02a05e thru 0x02e5ef (0x4592 bytes) 000037 1786103570.866702 write 0x0101b6 thru 0x012db2 (0x2bfd bytes) 000039 1786103570.868549 read 0x03fb73 thru 0x03ffff (0x48d bytes) 000040 1786103570.869720 mapread 0x024688 thru 0x03289b (0xe214 bytes) 000042 1786103570.872261 write 0x000340 thru 0x0075e0 (0x72a1 bytes) 000043 1786103570.873571 read 0x0272e9 thru 0x02a189 (0x2ea1 bytes) 000044 1786103570.874718 read 0x02bcb8 thru 0x02f687 (0x39d0 bytes) 000046 1786103570.876504 mapread 0x03ebff thru 0x03ffff (0x1401 bytes) 000047 1786103570.878273 trunc from 0x040000 to 0x003b2d (0x3c4d2 bytes) 000048 1786103570.901778 mapread 0x000d8f thru 0x003b2d (0x2d9f bytes) 000049 1786103570.902872 trunc from 0x003b2e to 0x006c2a (0x30fd bytes) 000052 1786103570.934695 write 0x02f9ff thru 0x03ae2b (0xb42d bytes) 000053 1786103570.937473 write 0x0169ce thru 0x019fe5 (0x3618 bytes) 000054 1786103570.977020 mapread 0x01e5cf thru 0x02975c (0xb18e bytes) 000056 1786103570.978838 read 0x014d08 thru 0x019708 (0x4a01 bytes) 000057 1786103570.980584 trunc from 0x03ae2c to 0x0091ba (0x31c71 bytes) 000058 1786103571.001396 trunc from 0x0091bb to 0x00b243 (0x2089 bytes) 000059 1786103571.028990 read 0x00acd4 thru 0x00b243 (0x570 bytes) 000060 1786103571.044469 read 0x000147 thru 0x00b0cd (0xaf87 bytes) 000061 1786103571.048548 mapread 0x005797 thru 0x00b243 (0x5aad bytes) 000062 1786103571.051220 read 0x000fec thru 0x00a00f (0x9024 bytes) 000063 1786103571.052968 read 0x00559c thru 0x00b243 (0x5ca8 bytes) 000064 1786103571.053782 mapwrite 0x009c9d thru 0x0173fa (0xd75e bytes) 000066 1786103571.237777 trunc from 0x0173fb to 0x01a2b8 (0x2ebe bytes) 000070 1786103571.281726 mapread 0x017b52 thru 0x01a2b8 (0x2767 bytes) truncating to largest ever: 0x3cb8d 000071 1786103571.299359 trunc from 0x01a2b9 to 0x03cb8c (0x228d4 bytes) 000073 1786103571.327210 MIRROR_EXTEND to 2 mirrors 000074 1786103572.247838 read 0x02059a thru 0x028836 (0x829d bytes) 000076 1786103572.263834 mapwrite 0x0377e1 thru 0x03f2ed (0x7b0d bytes) 000077 1786103572.485404 read 0x023ac9 thru 0x0311a4 (0xd6dc bytes) 000079 1786103572.501673 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000081 1786103572.724917 trunc from 0x03f2ee to 0x02c404 (0x12ee9 bytes) 000084 1786103572.749729 mapwrite 0x004eee thru 0x00a7dd (0x58f0 bytes) 000085 1786103572.816478 write 0x02ac8c thru 0x037cdd (0xd052 bytes) 000086 1786103572.821927 write 0x0032ce thru 0x00e906 (0xb639 bytes) 000090 1786103572.825533 read 0x000393 thru 0x00a2df (0x9f4d bytes) 000095 1786103572.836659 mapwrite 0x00c6c4 thru 0x019153 (0xca90 bytes) 000096 1786103572.978573 mapwrite 0x00301e thru 0x0097d0 (0x67b3 bytes) 000098 1786103573.059569 trunc from 0x037cde to 0x026f8b (0x10d52 bytes) 000101 1786103573.096944 write 0x023359 thru 0x02df4a (0xabf2 bytes) 000102 1786103573.117679 read 0x01bcd6 thru 0x0287ab (0xcad6 bytes) 000104 1786103573.133835 read 0x00aee7 thru 0x01660c (0xb726 bytes) 000105 1786103573.134547 trunc from 0x02df4b to 0x01810b (0x15e3f bytes) 000107 1786103573.158677 trunc from 0x01810c to 0x01792d (0x7de bytes) 000108 1786103573.184415 trunc from 0x01792e to 0x01d247 (0x591a bytes) 000111 1786103573.211390 mapread 0x0176b4 thru 0x018626 (0xf73 bytes) 000113 1786103573.226663 mapwrite 0x03de65 thru 0x03ffff (0x219b bytes) 000115 1786103573.327726 read 0x0350db thru 0x0358a0 (0x7c6 bytes) 000116 1786103573.332738 MIRROR_EXTEND to 2 mirrors 000119 1786103574.197934 write 0x026062 thru 0x030288 (0xa227 bytes) 000121 1786103574.280169 read 0x004a74 thru 0x004f2c (0x4b9 bytes) 000123 1786103574.288900 trunc from 0x040000 to 0x0070ad (0x38f52 bytes) 000124 1786103574.327792 mapwrite 0x033035 thru 0x03cfb1 (0x9f7d bytes) 000125 1786103574.454248 write 0x027f12 thru 0x02e072 (0x6161 bytes) 000127 1786103574.461765 trunc from 0x03cfb2 to 0x0206dd (0x1c8d4 bytes) 000130 1786103574.481326 mapread 0x00ba7a thru 0x014cc3 (0x924a bytes) 000131 1786103574.506549 write 0x00fbaf thru 0x013f37 (0x4389 bytes) 000134 1786103574.514184 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3f13c 000135 1786103575.306903 trunc from 0x0206de to 0x03f13b (0x1ea5e bytes) 000136 1786103575.431191 mapwrite 0x01d712 thru 0x02c376 (0xec65 bytes) 000138 1786103575.527517 mapwrite 0x0320e9 thru 0x03b651 (0x9569 bytes) 000139 1786103575.603013 write 0x03f07a thru 0x03ffff (0xf86 bytes) 000140 1786103575.603981 mapwrite 0x01d7e1 thru 0x025c9a (0x84ba bytes) 000141 1786103575.655537 trunc from 0x040000 to 0x01fad7 (0x20528 bytes) 000142 1786103575.674533 mapread 0x014853 thru 0x0186cc (0x3e7a bytes) 000143 1786103575.690262 read 0x0023ab thru 0x011b37 (0xf78d bytes) 000147 1786103575.703906 write 0x03bc78 thru 0x03ffff (0x4388 bytes) 000149 1786103575.706104 write 0x019bd2 thru 0x02890e (0xed3d bytes) 000151 1786103575.717521 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc21ea932 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7d06fd71 Mirror 3: 0x7d06fd71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000152 1786103576.501975 write 0x023215 thru 0x0295c8 (0x63b4 bytes) 000153 1786103576.614590 MIRROR_RESYNC 3 mirrors 000155 1786103577.561684 trunc from 0x040000 to 0x02641a (0x19be5 bytes) 000156 1786103577.621836 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2641b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2641b): Mirror 2: 0x699ebc62 (stale) Mirror 3: 0x699ebc62 (stale) CRC-32 checksum value for chunk [0, 0x2641b): Mirror 1: 0x699ebc62 Mirror 4: 0x699ebc62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2641b: skipped 000158 1786103578.466482 mapwrite 0x0015a0 thru 0x00c062 (0xaac3 bytes) 000160 1786103578.644226 read 0x01384e thru 0x021b87 (0xe33a bytes) 000161 1786103578.654269 trunc from 0x02641b to 0x01d11b (0x92ff bytes) 000162 1786103578.707481 trunc from 0x01d11c to 0x017f71 (0x51aa bytes) 000164 1786103578.747540 mapwrite 0x006172 thru 0x012895 (0xc724 bytes) 000165 1786103578.850335 MIRROR_RESYNC 4 mirrors 000166 1786103579.636832 mapread 0x016049 thru 0x017f71 (0x1f29 bytes) 000168 1786103579.648136 mapread 0x00d415 thru 0x014a74 (0x7660 bytes) 000169 1786103579.657573 mapread 0x012c32 thru 0x017f71 (0x5340 bytes) 000170 1786103579.659173 write 0x004953 thru 0x0102ca (0xb978 bytes) 000172 1786103579.748272 write 0x03db2b thru 0x03fcca (0x21a0 bytes) 000174 1786103579.752975 trunc from 0x03fccb to 0x033ce8 (0xbfe2 bytes) 000176 1786103579.778834 write 0x0224e5 thru 0x02e8af (0xc3cb bytes) 000178 1786103579.791705 mapwrite 0x033aff thru 0x039d03 (0x6205 bytes) 000179 1786103579.884640 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39d04): Mirror 2: 0xadb95d6a (stale) Mirror 4: 0xadb95d6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d04): Mirror 2: 0xadb95d6a (stale) Mirror 4: 0xadb95d6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d04: skipped truncating to largest ever: 0x3f4c7 000181 1786103580.253636 trunc from 0x039d04 to 0x03f4c6 (0x57c3 bytes) 000182 1786103580.298888 mapread 0x02b543 thru 0x02ebb9 (0x3677 bytes) 000185 1786103580.340858 trunc from 0x03f4c7 to 0x01e975 (0x20b51 bytes) 000186 1786103580.368796 trunc from 0x01e976 to 0x020447 (0x1ad2 bytes) 000188 1786103580.399493 mapwrite 0x01c49f thru 0x02346a (0x6fcc bytes) 000191 1786103580.511546 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2346b): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2346b): Mirror 2: 0xadfcc635 (stale) Mirror 4: 0xadfcc635 (stale) CRC-32 checksum value for chunk [0, 0x2346b): Mirror 1: 0x931c68c6 Mirror 5: 0x931c68c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2346b: skipped 000194 1786103581.115699 trunc from 0x02346b to 0x031df5 (0xe98b bytes) 000196 1786103581.219902 mapread 0x001613 thru 0x00e25c (0xcc4a bytes) 000198 1786103581.263210 write 0x0250cb thru 0x027455 (0x238b bytes) 000199 1786103581.276069 write 0x00c206 thru 0x01b0b5 (0xeeb0 bytes) 000200 1786103581.284811 mapwrite 0x02c201 thru 0x03afd0 (0xedd0 bytes) 000202 1786103581.430621 write 0x01a2eb thru 0x020dee (0x6b04 bytes) 000203 1786103581.442086 mapread 0x01507f thru 0x01bf9c (0x6f1e bytes) 000205 1786103581.443711 write 0x03c33a thru 0x03ffff (0x3cc6 bytes) 000206 1786103581.450297 write 0x029d7e thru 0x02a85c (0xadf bytes) 000207 1786103581.459973 trunc from 0x040000 to 0x0061ac (0x39e53 bytes) 000208 1786103581.499746 mapread 0x000106 thru 0x0061ac (0x60a7 bytes) 000210 1786103581.510600 mapwrite 0x02dee3 thru 0x02e837 (0x955 bytes) 000212 1786103581.649611 trunc from 0x02e838 to 0x039117 (0xa8e0 bytes) 000213 1786103581.677681 write 0x030744 thru 0x03b9bb (0xb278 bytes) 000216 1786103581.690389 MIRROR_RESYNC 4 mirrors 000217 1786103582.784613 mapread 0x006034 thru 0x00bcba (0x5c87 bytes) 000222 1786103582.803414 write 0x0109ec thru 0x010e73 (0x488 bytes) 000224 1786103582.861642 mapwrite 0x026d5c thru 0x027061 (0x306 bytes) 000225 1786103582.915078 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b9bc): Mirror 4: 0x97ae4b52 (stale) Mirror 5: 0x97ae4b52 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9bc): Mirror 4: 0x97ae4b52 (stale) Mirror 5: 0x97ae4b52 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9bc: skipped 000226 1786103583.226987 mapwrite 0x0212b1 thru 0x02f3a8 (0xe0f8 bytes) 000230 1786103583.366746 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b9bc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9bc): Mirror 4: 0x97ae4b52 (stale) Mirror 5: 0x97ae4b52 (stale) CRC-32 checksum value for chunk [0, 0x3b9bc): Mirror 1: 0x5ca78d2e Mirror 6: 0x5ca78d2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9bc: skipped 000231 1786103584.160951 mapwrite 0x004bf9 thru 0x010601 (0xba09 bytes) 000232 1786103584.392747 trunc from 0x03b9bc to 0x027125 (0x14896 bytes) 000234 1786103584.481518 write 0x0028e6 thru 0x00f5ed (0xcd08 bytes) 000235 1786103584.497101 MIRROR_RESYNC 4 mirrors 000236 1786103585.212248 write 0x015543 thru 0x01f4c5 (0x9f83 bytes) 000240 1786103585.285313 write 0x0216b4 thru 0x02ba66 (0xa3b3 bytes) 000244 1786103585.293500 write 0x03e3e4 thru 0x03ffff (0x1c1c bytes) 000250 1786103585.299517 mapwrite 0x0164b0 thru 0x02619b (0xfcec bytes) 000251 1786103585.360470 read 0x013943 thru 0x017bd2 (0x4290 bytes) 000252 1786103585.365320 mapwrite 0x03207f thru 0x03fcdd (0xdc5f bytes) 000253 1786103585.419969 trunc from 0x040000 to 0x017dae (0x28251 bytes) 000254 1786103585.437648 mapread 0x00b56d thru 0x017dae (0xc842 bytes) 000256 1786103585.451301 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17daf): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17daf): Mirror 4: 0xebf96561 (stale) Mirror 5: 0xebf96561 (stale) Mirror 6: 0xebf96561 (stale) CRC-32 checksum value for chunk [0, 0x17daf): Mirror 1: 0x60f81c0d Mirror 7: 0x60f81c0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17daf: skipped 000258 1786103586.109233 write 0x024029 thru 0x032119 (0xe0f1 bytes) 000259 1786103586.193792 mapread 0x009ca1 thru 0x0197cc (0xfb2c bytes) 000262 1786103586.259882 read 0x029330 thru 0x032119 (0x8dea bytes) 000263 1786103586.260840 trunc from 0x03211a to 0x007175 (0x2afa4 bytes) 000264 1786103586.330658 mapwrite 0x033b59 thru 0x03d1ad (0x9655 bytes) 000268 1786103586.421571 read 0x00a77d thru 0x014cca (0xa54e bytes) 000270 1786103586.432269 read 0x024498 thru 0x024674 (0x1dd bytes) 000272 1786103586.442005 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d1ae): Mirror 4: 0x658f4a5f (stale) Mirror 5: 0x658f4a5f (stale) Mirror 7: 0x39de78be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1ae): Mirror 4: 0x658f4a5f (stale) Mirror 5: 0x658f4a5f (stale) Mirror 7: 0x39de78be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1ae: skipped 000274 1786103586.799214 write 0x01a44b thru 0x022c37 (0x87ed bytes) 000276 1786103586.811825 trunc from 0x03d1ae to 0x005b37 (0x37676 bytes) 000277 1786103586.824880 trunc from 0x005b38 to 0x014d17 (0xf1e0 bytes) 000280 1786103586.845131 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x14d18): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14d18): Mirror 4: 0x639bfcd2 (stale) Mirror 5: 0x639bfcd2 (stale) Mirror 7: 0x639bfcd2 (stale) CRC-32 checksum value for chunk [0, 0x14d18): Mirror 1: 0x5e830f Mirror 8: 0x5e830f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14d18: skipped 000281 1786103587.734197 read 0x00dbae thru 0x014d17 (0x716a bytes) 000283 1786103587.778837 read 0x005ef7 thru 0x01443c (0xe546 bytes) 000284 1786103587.797626 write 0x01c4b5 thru 0x02b86d (0xf3b9 bytes) 000285 1786103587.884466 write 0x03393a thru 0x035ae2 (0x21a9 bytes) 000289 1786103587.888784 trunc from 0x035ae3 to 0x00f43c (0x266a6 bytes) 000290 1786103587.910321 write 0x015eb4 thru 0x01b328 (0x5475 bytes) 000291 1786103587.915959 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b329): Mirror 4: 0xba4528d7 (stale) Mirror 5: 0xba4528d7 (stale) Mirror 8: 0xdf1ff2aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b329): Mirror 4: 0xba4528d7 (stale) Mirror 5: 0xba4528d7 (stale) Mirror 8: 0xdf1ff2aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b329: skipped 000293 1786103588.238238 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b329): Mirror 4: 0xba4528d7 (stale) Mirror 8: 0xdf1ff2aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b329): Mirror 4: 0xba4528d7 (stale) Mirror 8: 0xdf1ff2aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b329: skipped 000294 1786103588.502678 mapwrite 0x0101a1 thru 0x01b662 (0xb4c2 bytes) 000295 1786103588.660930 mapread 0x014307 thru 0x01b662 (0x735c bytes) 000297 1786103588.677753 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b663): Mirror 8: 0x78915d71 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b663): Mirror 8: 0x78915d71 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b663: skipped 000298 1786103588.978806 mapread 0x018302 thru 0x018890 (0x58f bytes) 000299 1786103588.990164 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000300 1786103589.104571 read 0x0152be thru 0x016a79 (0x17bc bytes) 000301 1786103589.110380 mapwrite 0x007a76 thru 0x01714a (0xf6d5 bytes) 000304 1786103589.175567 mapread 0x003f35 thru 0x013d24 (0xfdf0 bytes) 000305 1786103589.179801 mapwrite 0x039c16 thru 0x03b6ad (0x1a98 bytes) 000306 1786103589.235630 read 0x03a04c thru 0x03b6ad (0x1662 bytes) 000308 1786103589.237410 read 0x01a7c7 thru 0x01b0e5 (0x91f bytes) 000309 1786103589.246401 mapwrite 0x01d8aa thru 0x023c45 (0x639c bytes) 000310 1786103589.296995 read 0x027434 thru 0x0290cc (0x1c99 bytes) 000313 1786103589.300349 write 0x02ba57 thru 0x03a2bc (0xe866 bytes) 000316 1786103589.304791 write 0x025149 thru 0x027488 (0x2340 bytes) 000317 1786103589.305878 trunc from 0x03b6ae to 0x01488a (0x26e23 bytes) 000318 1786103589.343410 mapread 0x010ccb thru 0x01488a (0x3bc0 bytes) 000319 1786103589.346558 mapread 0x00020e thru 0x0039d8 (0x37cb bytes) 000321 1786103589.352694 trunc from 0x01488b to 0x021383 (0xcaf9 bytes) 000323 1786103589.395215 mapread 0x0070e6 thru 0x00e3a7 (0x72c2 bytes) 000325 1786103589.399955 read 0x00b951 thru 0x01206f (0x671f bytes) 000326 1786103589.401413 mapread 0x01143f thru 0x01ecbd (0xd87f bytes) 000327 1786103589.416476 mapwrite 0x026edf thru 0x02c404 (0x5526 bytes) 000328 1786103589.503506 read 0x0188ac thru 0x024098 (0xb7ed bytes) 000329 1786103589.505813 read 0x019a22 thru 0x022330 (0x890f bytes) 000330 1786103589.507608 read 0x0117a1 thru 0x011ad0 (0x330 bytes) 000331 1786103589.509477 write 0x011fad thru 0x0197c4 (0x7818 bytes) 000333 1786103589.514670 trunc from 0x02c405 to 0x020848 (0xbbbc bytes) 000335 1786103589.538545 trunc from 0x020849 to 0x00415d (0x1c6eb bytes) 000336 1786103589.554592 mapwrite 0x00bc55 thru 0x00d868 (0x1c14 bytes) 000339 1786103589.626592 mapwrite 0x016128 thru 0x01f7e3 (0x96bc bytes) 000344 1786103589.705394 write 0x0356c7 thru 0x03ffff (0xa939 bytes) 000345 1786103589.706884 write 0x01e6e2 thru 0x028be5 (0xa504 bytes) 000346 1786103589.712770 trunc from 0x040000 to 0x03225a (0xdda5 bytes) 000347 1786103589.733802 mapread 0x02a6fb thru 0x03225a (0x7b60 bytes) 000348 1786103589.748188 mapwrite 0x0347a3 thru 0x0380aa (0x3908 bytes) 000349 1786103589.823262 read 0x01c54f thru 0x022efb (0x69ad bytes) 000350 1786103589.824389 trunc from 0x0380ab to 0x007c7f (0x3042b bytes) 000351 1786103589.840488 trunc from 0x007c80 to 0x0380c4 (0x30445 bytes) 000352 1786103589.857009 write 0x02e73b thru 0x03c506 (0xddcc bytes) 000353 1786103589.866469 trunc from 0x03c507 to 0x02bfd1 (0x10535 bytes) 000354 1786103589.885283 read 0x025d81 thru 0x02bfd1 (0x6251 bytes) 000360 1786103589.890267 mapwrite 0x010bad thru 0x01fb94 (0xefe8 bytes) 000362 1786103589.935519 read 0x022877 thru 0x02bfd1 (0x975b bytes) 000363 1786103589.937188 trunc from 0x02bfd2 to 0x02d992 (0x19c1 bytes) 000364 1786103589.955661 write 0x013f9b thru 0x0146b8 (0x71e bytes) 000365 1786103589.956809 trunc from 0x02d993 to 0x00140c (0x2c586 bytes) 000366 1786103589.980907 mapread 0x000751 thru 0x00140c (0xcbc bytes) 000367 1786103589.981986 trunc from 0x00140d to 0x004e56 (0x3a4a bytes) 000373 1786103590.014823 read 0x003a0b thru 0x004e56 (0x144c bytes) 000374 1786103590.022448 mapread 0x00165b thru 0x004e56 (0x37fc bytes) 000376 1786103590.037470 trunc from 0x004e57 to 0x000132 (0x4d24 bytes) 000377 1786103590.076306 MIRROR_EXTEND to 2 mirrors 000378 1786103590.621391 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000381 1786103590.745017 trunc from 0x000133 to 0x00a06a (0x9f38 bytes) 000382 1786103590.804777 write 0x005e7a thru 0x00b5fd (0x5784 bytes) 000385 1786103590.812502 write 0x01eae1 thru 0x026eeb (0x840b bytes) 000386 1786103590.814655 read 0x0061fa thru 0x01538a (0xf191 bytes) 000387 1786103590.824358 write 0x014be1 thru 0x01c6ba (0x7ada bytes) 000390 1786103590.831868 write 0x03580e thru 0x03ffff (0xa7f2 bytes) 000393 1786103590.838574 write 0x017b1e thru 0x023d41 (0xc224 bytes) 000395 1786103590.841631 write 0x01f257 thru 0x02d31f (0xe0c9 bytes) 000396 1786103590.847558 mapread 0x033feb thru 0x0382d7 (0x42ed bytes) 000397 1786103590.857936 mapread 0x0347cc thru 0x03ffff (0xb834 bytes) 000398 1786103590.859937 mapwrite 0x01b49b thru 0x01d454 (0x1fba bytes) 000399 1786103590.933532 read 0x02e0d5 thru 0x034dd6 (0x6d02 bytes) 000401 1786103590.946587 trunc from 0x040000 to 0x016306 (0x29cf9 bytes) 000404 1786103590.973590 read 0x00a150 thru 0x016306 (0xc1b7 bytes) 000405 1786103590.974424 read 0x0055db thru 0x0065f4 (0x101a bytes) 000407 1786103590.985337 mapwrite 0x000250 thru 0x005c6a (0x5a1b bytes) 000408 1786103591.081964 MIRROR_EXTEND to 2 mirrors 000410 1786103591.734318 mapread 0x001fc2 thru 0x004d09 (0x2d48 bytes) 000411 1786103591.752233 mapwrite 0x02ff98 thru 0x038191 (0x81fa bytes) 000412 1786103591.902414 trunc from 0x038192 to 0x02a128 (0xe069 bytes) 000414 1786103591.921804 mapread 0x022c97 thru 0x0299b0 (0x6d1a bytes) 000416 1786103591.932524 mapread 0x021a5d thru 0x0278f8 (0x5e9c bytes) 000418 1786103591.934924 trunc from 0x02a129 to 0x018073 (0x120b5 bytes) 000419 1786103591.953676 trunc from 0x018074 to 0x02fb0a (0x17a97 bytes) 000420 1786103591.972604 trunc from 0x02fb0b to 0x02299b (0xd16f bytes) 000422 1786103591.986993 mapread 0x016e20 thru 0x01d79f (0x6980 bytes) 000425 1786103592.000524 mapread 0x009de6 thru 0x00ba4e (0x1c69 bytes) 000426 1786103592.015276 MIRROR_RESYNC 2 mirrors 000427 1786103592.596984 trunc from 0x02299c to 0x008081 (0x1a91a bytes) 000429 1786103592.662368 mapread 0x00101e thru 0x008081 (0x7064 bytes) 000433 1786103592.706399 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x8082): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8082): Mirror 2: 0x2920c506 (stale) CRC-32 checksum value for chunk [0, 0x8082): Mirror 1: 0x2920c506 Mirror 3: 0x2920c506 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8082: skipped 000435 1786103593.506252 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x8082): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8082): Mirror 2: 0x2920c506 (stale) CRC-32 checksum value for chunk [0, 0x8082): Mirror 1: 0x2920c506 Mirror 3: 0x2920c506 Mirror 4: 0x2920c506 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8082: skipped 000436 1786103594.286990 trunc from 0x008082 to 0x0309f4 (0x28973 bytes) 000437 1786103594.457927 trunc from 0x0309f5 to 0x01d7d2 (0x13222 bytes) 000438 1786103594.505487 trunc from 0x01d7d3 to 0x01bdb5 (0x1a1d bytes) 000441 1786103594.523013 mapwrite 0x03b85b thru 0x03ffff (0x47a5 bytes) 000443 1786103594.614349 mapwrite 0x03113c thru 0x03fed1 (0xed96 bytes) 000444 1786103594.721415 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x27e8f168 (stale) Mirror 3: 0x7c053311 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x27e8f168 (stale) Mirror 3: 0x7c053311 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000445 1786103595.168648 mapwrite 0x01d6e1 thru 0x020d25 (0x3645 bytes) 000447 1786103595.281294 mapread 0x001f34 thru 0x00a6c0 (0x878d bytes) 000448 1786103595.292244 write 0x018833 thru 0x0228ff (0xa0cd bytes) 000450 1786103595.295772 trunc from 0x040000 to 0x0087fb (0x37804 bytes) 000452 1786103595.329213 trunc from 0x0087fc to 0x022e80 (0x1a685 bytes) 000453 1786103595.353909 trunc from 0x022e81 to 0x0016ca (0x217b6 bytes) 000454 1786103595.371315 read 0x000223 thru 0x0016ca (0x14a8 bytes) 000455 1786103595.376388 mapread 0x000d7e thru 0x0016ca (0x94d bytes) 000457 1786103595.381891 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16cb): Mirror 3: 0x9c7fac92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16cb): Mirror 3: 0x9c7fac92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16cb: skipped 000459 1786103595.621484 mapread 0x000ffc thru 0x0016ca (0x6cf bytes) 000461 1786103595.638382 trunc from 0x0016cb to 0x024d14 (0x2364a bytes) 000462 1786103595.673928 write 0x03e4a0 thru 0x03ffff (0x1b60 bytes) 000463 1786103595.675144 mapwrite 0x01c9b4 thru 0x0237c6 (0x6e13 bytes) 000464 1786103595.801808 write 0x03aa7e thru 0x03ffff (0x5582 bytes) 000467 1786103595.831660 mapwrite 0x0249b4 thru 0x028e59 (0x44a6 bytes) 000469 1786103595.951401 mapread 0x02ec22 thru 0x034ee7 (0x62c6 bytes) 000470 1786103595.990248 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7c053311 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x71838863 Mirror 4: 0x71838863 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000471 1786103596.867246 write 0x0234f6 thru 0x0316cf (0xe1da bytes) 000475 1786103597.054132 write 0x02924f thru 0x02e317 (0x50c9 bytes) 000477 1786103597.064985 trunc from 0x040000 to 0x02cefb (0x13104 bytes) 000478 1786103597.102117 write 0x035a4d thru 0x03a0a9 (0x465d bytes) 000479 1786103597.105593 mapwrite 0x02284b thru 0x02d7c0 (0xaf76 bytes) 000480 1786103597.223960 mapread 0x0160ca thru 0x01a326 (0x425d bytes) 000482 1786103597.249745 trunc from 0x03a0aa to 0x01d964 (0x1c745 bytes) 000483 1786103597.272652 read 0x00bb21 thru 0x00ca0e (0xeee bytes) 000484 1786103597.287907 read 0x005e8e thru 0x00f5b5 (0x9728 bytes) 000488 1786103597.307206 trunc from 0x01d965 to 0x012d3e (0xac26 bytes) 000490 1786103597.341705 MIRROR_RESYNC 3 mirrors 000494 1786103598.105214 write 0x01e6f9 thru 0x0204c9 (0x1dd1 bytes) 000495 1786103598.184604 mapread 0x0021bc thru 0x00a49f (0x82e4 bytes) 000497 1786103598.207135 mapread 0x017a2b thru 0x01dd7b (0x6351 bytes) 000500 1786103598.221307 MIRROR_RESYNC 3 mirrors 000502 1786103598.901635 write 0x00ad81 thru 0x0116b0 (0x6930 bytes) 000503 1786103599.086238 write 0x003320 thru 0x008c8d (0x596e bytes) 000504 1786103599.098778 mapwrite 0x031631 thru 0x0358ca (0x429a bytes) 000505 1786103599.213377 read 0x004af4 thru 0x005809 (0xd16 bytes) 000506 1786103599.214155 mapread 0x018e74 thru 0x01d61c (0x47a9 bytes) 000507 1786103599.223558 mapwrite 0x012c22 thru 0x019a99 (0x6e78 bytes) 000508 1786103599.257836 write 0x008ff0 thru 0x00dda3 (0x4db4 bytes) 000509 1786103599.259395 write 0x00e295 thru 0x01168c (0x33f8 bytes) 000514 1786103599.268155 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x358cb): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358cb): Mirror 3: 0x6344c33a (stale) Mirror 4: 0x6344c33a (stale) CRC-32 checksum value for chunk [0, 0x358cb): Mirror 1: 0x37e15d49 Mirror 5: 0x37e15d49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358cb: skipped 000523 1786103599.997349 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x358cb): Mirror 4: 0x6344c33a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358cb): Mirror 4: 0x6344c33a (stale) CRC-32 checksum value for chunk [0, 0x358cb): Mirror 1: 0x37e15d49 Mirror 5: 0x37e15d49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358cb: skipped 000524 1786103600.488804 trunc from 0x0358cb to 0x039d33 (0x4469 bytes) 000526 1786103600.610351 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39d34): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d34): Mirror 4: 0x79888028 (stale) Mirror 5: 0x19983c (stale) CRC-32 checksum value for chunk [0, 0x39d34): Mirror 1: 0x19983c Mirror 6: 0x19983c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d34: skipped 000528 1786103601.603417 read 0x02b095 thru 0x02cd0e (0x1c7a bytes) 000529 1786103601.612861 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39d34): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d34): Mirror 4: 0x79888028 (stale) Mirror 5: 0x19983c (stale) CRC-32 checksum value for chunk [0, 0x39d34): Mirror 1: 0x19983c Mirror 6: 0x19983c Mirror 7: 0x19983c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d34: skipped 000531 1786103602.506417 write 0x01bf7a thru 0x022458 (0x64df bytes) 000534 1786103602.594285 mapread 0x028039 thru 0x029c32 (0x1bfa bytes) 000535 1786103602.607504 trunc from 0x039d34 to 0x00bdd0 (0x2df63 bytes) 000538 1786103602.637652 trunc from 0x00bdd1 to 0x035850 (0x29a80 bytes) 000540 1786103602.653946 mapwrite 0x00ffee thru 0x0188e1 (0x88f4 bytes) 000541 1786103602.727772 trunc from 0x035851 to 0x02499b (0x10eb5 bytes) 000542 1786103602.762623 trunc from 0x02499c to 0x02af2e (0x6593 bytes) 000547 1786103602.798544 mapwrite 0x034c35 thru 0x03982d (0x4bf9 bytes) 000548 1786103602.856567 mapwrite 0x01da02 thru 0x028e5e (0xb45d bytes) 000551 1786103602.916553 read 0x02fc1d thru 0x0316ed (0x1ad1 bytes) 000552 1786103602.917282 read 0x032c9a thru 0x036e62 (0x41c9 bytes) 000553 1786103602.922616 mapwrite 0x008692 thru 0x0113e4 (0x8d53 bytes) 000554 1786103603.000844 read 0x0280d3 thru 0x0357f5 (0xd723 bytes) 000555 1786103603.009559 trunc from 0x03982e to 0x020dba (0x18a73 bytes) 000556 1786103603.045377 MIRROR_RESYNC 5 mirrors 000557 1786103603.974253 mapread 0x003b83 thru 0x0117f3 (0xdc71 bytes) 000560 1786103604.013203 write 0x014f53 thru 0x021fcf (0xd07d bytes) 000562 1786103604.101783 MIRROR_RESYNC 5 mirrors 000566 1786103605.035657 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000568 1786103605.408400 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000569 1786103605.710597 read 0x011632 thru 0x012fec (0x19bb bytes) 000571 1786103605.719268 write 0x001426 thru 0x00f42d (0xe008 bytes) 000572 1786103605.806948 mapread 0x0191e9 thru 0x01b567 (0x237f bytes) 000576 1786103605.831981 write 0x02a103 thru 0x02c1d1 (0x20cf bytes) 000580 1786103605.836507 read 0x02a03a thru 0x02c1d1 (0x2198 bytes) 000581 1786103605.838750 trunc from 0x02c1d2 to 0x038444 (0xc273 bytes) 000582 1786103605.866833 mapwrite 0x00931e thru 0x00a156 (0xe39 bytes) 000584 1786103605.922394 read 0x00c7a5 thru 0x01b424 (0xec80 bytes) 000585 1786103605.943619 write 0x02d76e thru 0x02d8e4 (0x177 bytes) 000587 1786103605.945795 trunc from 0x038445 to 0x00a533 (0x2df11 bytes) 000589 1786103605.962251 trunc from 0x00a534 to 0x0272f5 (0x1cdc2 bytes) 000590 1786103605.980746 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x272f6): Mirror 6: 0x577aff5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x272f6): Mirror 6: 0x577aff5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x272f6: skipped 000595 1786103606.234928 mapwrite 0x008276 thru 0x014e60 (0xcbeb bytes) 000596 1786103606.352628 trunc from 0x0272f6 to 0x031853 (0xa55e bytes) 000597 1786103606.370580 mapwrite 0x020e9c thru 0x02aaef (0x9c54 bytes) 000598 1786103606.449604 mapwrite 0x001ee9 thru 0x007db7 (0x5ecf bytes) 000601 1786103606.537417 MIRROR_RESYNC 2 mirrors 000602 1786103607.242414 mapwrite 0x026bc6 thru 0x02d457 (0x6892 bytes) 000603 1786103607.430365 trunc from 0x031854 to 0x017670 (0x1a1e3 bytes) 000605 1786103607.476310 trunc from 0x017671 to 0x03485e (0x1d1ee bytes) 000606 1786103607.497067 mapread 0x01c0e6 thru 0x0203b9 (0x42d4 bytes) 000607 1786103607.508946 trunc from 0x03485f to 0x011cd3 (0x22b8b bytes) 000609 1786103607.536322 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x11cd4): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11cd4): Mirror 6: 0xc1ddb198 (stale) CRC-32 checksum value for chunk [0, 0x11cd4): Mirror 1: 0xc1ddb198 Mirror 7: 0xc1ddb198 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11cd4: skipped 000612 1786103608.490789 write 0x0285fd thru 0x031d65 (0x9769 bytes) 000613 1786103608.579267 read 0x021aeb thru 0x02e7f4 (0xcd0a bytes) 000614 1786103608.594673 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31d66): Mirror 7: 0xf2e2f8e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d66): Mirror 7: 0xf2e2f8e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d66: skipped 000617 1786103608.904843 write 0x002a4b thru 0x00e2ab (0xb861 bytes) 000620 1786103608.919718 trunc from 0x031d66 to 0x007ef8 (0x29e6d bytes) 000621 1786103608.936392 write 0x0168cf thru 0x01f2cf (0x8a01 bytes) 000624 1786103608.941157 trunc from 0x01f2d0 to 0x00a28e (0x15041 bytes) 000625 1786103608.964434 trunc from 0x00a28f to 0x001ceb (0x85a3 bytes) 000626 1786103608.984437 trunc from 0x001cec to 0x03c15a (0x3a46f bytes) 000627 1786103609.004297 write 0x00db34 thru 0x01d18e (0xf65b bytes) 000630 1786103609.023315 mapread 0x015013 thru 0x01b12a (0x6118 bytes) 000631 1786103609.027220 read 0x014b03 thru 0x015f69 (0x1467 bytes) 000632 1786103609.027895 mapwrite 0x00cc2a thru 0x0152c8 (0x869f bytes) 000634 1786103609.111440 MIRROR_RESYNC 2 mirrors 000635 1786103610.134717 mapwrite 0x021e90 thru 0x029a87 (0x7bf8 bytes) 000636 1786103610.341523 mapread 0x00ea86 thru 0x01b564 (0xcadf bytes) 000637 1786103610.365792 mapread 0x02f247 thru 0x03c15a (0xcf14 bytes) 000639 1786103610.404856 trunc from 0x03c15b to 0x0164f4 (0x25c66 bytes) 000641 1786103610.493310 MIRROR_RESYNC 2 mirrors 000642 1786103611.268821 write 0x013fcf thru 0x01bfa8 (0x7fda bytes) 000643 1786103611.352421 mapread 0x010e8b thru 0x013c46 (0x2dbc bytes) 000644 1786103611.372884 mapread 0x018697 thru 0x01bef0 (0x385a bytes) 000647 1786103611.380324 MIRROR_RESYNC 2 mirrors 000648 1786103611.942457 write 0x03f3be thru 0x03ffff (0xc42 bytes) 000650 1786103612.030920 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000651 1786103612.180250 read 0x0173ed thru 0x01a506 (0x311a bytes) 000652 1786103612.190663 MIRROR_EXTEND to 2 mirrors 000653 1786103613.052701 write 0x03be22 thru 0x03f5d7 (0x37b6 bytes) 000656 1786103613.199020 mapwrite 0x0212dc thru 0x02c371 (0xb096 bytes) 000657 1786103613.276132 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000658 1786103613.398244 mapread 0x03bf24 thru 0x03ffff (0x40dc bytes) 000659 1786103613.432720 trunc from 0x040000 to 0x025a4a (0x1a5b5 bytes) 000660 1786103613.481511 trunc from 0x025a4b to 0x024fa1 (0xaa9 bytes) 000662 1786103613.494836 write 0x03f442 thru 0x03ffff (0xbbe bytes) 000664 1786103613.497161 mapwrite 0x01f64d thru 0x026c3b (0x75ef bytes) 000666 1786103613.557817 read 0x03d518 thru 0x03ffff (0x2ae8 bytes) 000669 1786103613.569612 write 0x00aab9 thru 0x013f75 (0x94bd bytes) 000670 1786103613.582447 mapwrite 0x034e90 thru 0x03972c (0x489d bytes) 000671 1786103613.657142 mapread 0x014cef thru 0x0154af (0x7c1 bytes) 000672 1786103613.673350 write 0x010046 thru 0x016771 (0x672c bytes) 000673 1786103613.690349 MIRROR_EXTEND to 2 mirrors 000675 1786103614.600475 read 0x02ac85 thru 0x03214e (0x74ca bytes) 000677 1786103614.608396 read 0x00be1c thru 0x011c3a (0x5e1f bytes) 000678 1786103614.629611 MIRROR_RESYNC 2 mirrors 000679 1786103614.986620 write 0x022ecb thru 0x028fa0 (0x60d6 bytes) 000680 1786103615.071932 mapread 0x03b547 thru 0x03ffff (0x4ab9 bytes) 000681 1786103615.093023 read 0x015e77 thru 0x024179 (0xe303 bytes) 000685 1786103615.112664 write 0x013f23 thru 0x01cf44 (0x9022 bytes) 000687 1786103615.121427 mapread 0x00bc54 thru 0x014239 (0x85e6 bytes) 000691 1786103615.142377 read 0x01d6b2 thru 0x02a931 (0xd280 bytes) 000692 1786103615.172488 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000693 1786103615.393312 write 0x00292b thru 0x012398 (0xfa6e bytes) 000694 1786103615.404427 trunc from 0x040000 to 0x01a895 (0x2576a bytes) 000695 1786103615.424486 trunc from 0x01a896 to 0x006b77 (0x13d1e bytes) 000696 1786103615.442677 read 0x0053dd thru 0x006b77 (0x179b bytes) 000698 1786103615.451484 MIRROR_EXTEND to 2 mirrors 000699 1786103616.152791 trunc from 0x006b78 to 0x02eb93 (0x2801c bytes) 000700 1786103616.340217 trunc from 0x02eb94 to 0x0213e5 (0xd7ae bytes) 000705 1786103616.387492 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000707 1786103616.536364 trunc from 0x0213e6 to 0x014bbf (0xc826 bytes) 000711 1786103616.571787 mapwrite 0x00c344 thru 0x01a683 (0xe340 bytes) 000712 1786103616.745780 trunc from 0x01a684 to 0x03b6d7 (0x21054 bytes) 000714 1786103616.766911 mapread 0x009ad9 thru 0x00d22f (0x3757 bytes) 000715 1786103616.769261 write 0x03cef9 thru 0x03ffff (0x3107 bytes) 000719 1786103616.773687 mapread 0x01bfe8 thru 0x027832 (0xb84b bytes) 000720 1786103616.813884 MIRROR_EXTEND to 2 mirrors 000721 1786103617.800996 trunc from 0x040000 to 0x0220b0 (0x1df4f bytes) 000722 1786103617.896400 mapread 0x00dbcb thru 0x00df22 (0x358 bytes) 000724 1786103617.904240 mapread 0x00f987 thru 0x01dccf (0xe349 bytes) 000726 1786103617.932517 trunc from 0x0220b1 to 0x02a84b (0x879b bytes) 000727 1786103617.958668 write 0x0350d7 thru 0x03988b (0x47b5 bytes) 000728 1786103617.960303 write 0x006b66 thru 0x013bbb (0xd056 bytes) 000729 1786103617.968989 write 0x01621a thru 0x01a94b (0x4732 bytes) 000731 1786103617.971237 mapread 0x002e2a thru 0x007898 (0x4a6f bytes) 000733 1786103617.981592 read 0x01daa6 thru 0x027f6b (0xa4c6 bytes) 000734 1786103617.991955 trunc from 0x03988c to 0x029ad7 (0xfdb4 bytes) 000735 1786103618.010365 read 0x0087c3 thru 0x008c09 (0x447 bytes) 000736 1786103618.012194 mapwrite 0x039798 thru 0x03c9d7 (0x3240 bytes) 000737 1786103618.086330 trunc from 0x03c9d8 to 0x03d2fe (0x927 bytes) 000738 1786103618.102753 trunc from 0x03d2ff to 0x003a72 (0x3988c bytes) 000739 1786103618.131217 read 0x001ddc thru 0x003a72 (0x1c97 bytes) 000740 1786103618.134610 mapwrite 0x038a4d thru 0x03ffff (0x75b3 bytes) 000741 1786103618.249486 trunc from 0x040000 to 0x037d94 (0x826b bytes) 000742 1786103618.273495 trunc from 0x037d95 to 0x022d60 (0x15034 bytes) 000743 1786103618.300994 mapread 0x002e6f thru 0x010e70 (0xe002 bytes) 000744 1786103618.302355 mapread 0x014444 thru 0x01a770 (0x632d bytes) 000746 1786103618.321825 MIRROR_RESYNC 2 mirrors 000747 1786103619.102773 read 0x017a92 thru 0x01c907 (0x4e76 bytes) 000749 1786103619.120320 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000751 1786103619.266858 trunc from 0x022d61 to 0x00185b (0x21505 bytes) 000752 1786103619.316192 trunc from 0x00185c to 0x03322b (0x319d0 bytes) 000753 1786103619.332605 read 0x014865 thru 0x023467 (0xec03 bytes) 000756 1786103619.346177 write 0x0122eb thru 0x01e636 (0xc34c bytes) 000757 1786103619.352689 mapwrite 0x006abe thru 0x0102a7 (0x97ea bytes) 000758 1786103619.419955 write 0x0340b8 thru 0x03fe0b (0xbd54 bytes) 000759 1786103619.421449 trunc from 0x03fe0c to 0x03980e (0x65fd bytes) 000761 1786103619.435603 mapread 0x011ebc thru 0x0151c2 (0x3307 bytes) 000764 1786103619.442509 write 0x012bd9 thru 0x01f9f2 (0xce1a bytes) 000766 1786103619.449626 MIRROR_EXTEND to 2 mirrors 000769 1786103620.345977 mapread 0x024954 thru 0x02c70b (0x7db8 bytes) 000770 1786103620.369008 trunc from 0x03980f to 0x023399 (0x16475 bytes) 000771 1786103620.494728 mapread 0x00b475 thru 0x00da9a (0x2626 bytes) 000772 1786103620.509740 read 0x01039e thru 0x017098 (0x6cfb bytes) 000773 1786103620.516062 trunc from 0x02339a to 0x01cdb7 (0x65e2 bytes) 000775 1786103620.540344 write 0x025180 thru 0x030689 (0xb50a bytes) 000780 1786103620.549893 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3068a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3068a): Mirror 2: 0x472dd3a5 (stale) CRC-32 checksum value for chunk [0, 0x3068a): Mirror 1: 0xba483453 Mirror 3: 0xba483453 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3068a: skipped 000781 1786103621.686934 read 0x00a431 thru 0x0196c6 (0xf296 bytes) 000784 1786103621.715222 trunc from 0x03068a to 0x0116bd (0x1efcc bytes) 000785 1786103621.835315 trunc from 0x0116be to 0x002279 (0xf444 bytes) 000788 1786103621.857161 trunc from 0x00227a to 0x03ef28 (0x3ccaf bytes) 000789 1786103621.877334 mapwrite 0x02250b thru 0x023f2f (0x1a25 bytes) 000791 1786103621.961385 mapwrite 0x031a04 thru 0x033733 (0x1d30 bytes) 000792 1786103622.040899 read 0x013352 thru 0x01ec4b (0xb8fa bytes) 000793 1786103622.060394 read 0x0147c3 thru 0x020d88 (0xc5c6 bytes) 000794 1786103622.062362 mapread 0x01367a thru 0x01bc41 (0x85c8 bytes) 000795 1786103622.065744 mapwrite 0x03f662 thru 0x03ffff (0x99e bytes) 000797 1786103622.194370 trunc from 0x040000 to 0x030e5a (0xf1a5 bytes) 000799 1786103622.219522 trunc from 0x030e5b to 0x0159b2 (0x1b4a8 bytes) 000800 1786103622.237623 read 0x000ab1 thru 0x00ed0e (0xe25e bytes) 000802 1786103622.239504 mapwrite 0x0196b9 thru 0x022b3d (0x9485 bytes) 000803 1786103622.304987 trunc from 0x022b3e to 0x031b05 (0xefc8 bytes) 000804 1786103622.320229 read 0x02ee68 thru 0x031b05 (0x2c9e bytes) 000805 1786103622.328897 trunc from 0x031b06 to 0x0185ab (0x1955a bytes) 000810 1786103622.351854 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x185ac): Mirror 2: 0xf0439d76 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x185ac): Mirror 2: 0xf0439d76 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x185ac: skipped 000811 1786103622.570336 mapread 0x009b40 thru 0x017435 (0xd8f6 bytes) 000812 1786103622.596018 trunc from 0x0185ac to 0x00aed0 (0xd6db bytes) 000813 1786103622.608361 mapwrite 0x0293b3 thru 0x02f494 (0x60e2 bytes) 000814 1786103622.670375 mapwrite 0x02a726 thru 0x033eca (0x97a5 bytes) 000815 1786103622.754984 read 0x01f168 thru 0x027acd (0x8966 bytes) 000816 1786103622.763631 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33ecb): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ecb): Mirror 2: 0x8fc9ae76 (stale) CRC-32 checksum value for chunk [0, 0x33ecb): Mirror 1: 0xe9fbf1ce Mirror 3: 0xe9fbf1ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ecb: skipped 000819 1786103623.634349 mapwrite 0x025a54 thru 0x02cb9e (0x714b bytes) 000823 1786103623.925714 read 0x02870c thru 0x028ce4 (0x5d9 bytes) 000824 1786103623.932137 MIRROR_RESYNC 3 mirrors 000826 1786103625.265328 mapread 0x0323df thru 0x033eca (0x1aec bytes) 000829 1786103625.275091 mapread 0x012e16 thru 0x01f005 (0xc1f0 bytes) 000830 1786103625.296897 mapread 0x029de7 thru 0x02a580 (0x79a bytes) 000831 1786103625.331104 write 0x03d49e thru 0x03ffff (0x2b62 bytes) 000832 1786103625.435389 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd43d4835 (stale) Mirror 3: 0xd43d4835 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf9b8087d Mirror 4: 0xf9b8087d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000835 1786103626.377481 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd43d4835 (stale) Mirror 3: 0xd43d4835 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf9b8087d Mirror 4: 0xf9b8087d Mirror 5: 0xf9b8087d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000841 1786103627.302793 read 0x02cf60 thru 0x033092 (0x6133 bytes) 000842 1786103627.324775 read 0x023163 thru 0x024687 (0x1525 bytes) 000843 1786103627.352835 mapwrite 0x0173eb thru 0x022d85 (0xb99b bytes) 000845 1786103627.795252 mapread 0x0117ab thru 0x014a16 (0x326c bytes) 000847 1786103627.811201 mapwrite 0x0294ff thru 0x037f7b (0xea7d bytes) 000848 1786103627.909897 trunc from 0x040000 to 0x01db36 (0x224c9 bytes) 000849 1786103627.938419 mapwrite 0x024153 thru 0x024973 (0x821 bytes) 000850 1786103628.003346 trunc from 0x024974 to 0x006b1c (0x1de57 bytes) 000851 1786103628.016816 write 0x01671c thru 0x01bdd2 (0x56b7 bytes) 000852 1786103628.019141 read 0x00cde8 thru 0x016982 (0x9b9b bytes) 000853 1786103628.029020 trunc from 0x01bdd3 to 0x03e3c9 (0x225f7 bytes) 000855 1786103628.040559 mapwrite 0x0218e8 thru 0x028ed8 (0x75f1 bytes) 000856 1786103628.127464 mapwrite 0x03b353 thru 0x03ffff (0x4cad bytes) 000857 1786103628.209017 read 0x0328b5 thru 0x039a3f (0x718b bytes) 000858 1786103628.209762 mapwrite 0x02fea8 thru 0x032e1a (0x2f73 bytes) 000859 1786103628.269555 mapread 0x027736 thru 0x029e65 (0x2730 bytes) 000862 1786103628.280189 mapwrite 0x024eb6 thru 0x02e24a (0x9395 bytes) 000865 1786103628.383448 write 0x03bfc0 thru 0x03d10b (0x114c bytes) 000869 1786103628.388312 write 0x034568 thru 0x03513a (0xbd3 bytes) 000872 1786103628.392722 read 0x02c42b thru 0x03b59f (0xf175 bytes) 000873 1786103628.393951 trunc from 0x040000 to 0x003d94 (0x3c26b bytes) 000874 1786103628.416717 MIRROR_RESYNC 5 mirrors 000875 1786103629.341461 MIRROR_EXTEND to 6 mirrors 000876 1786103630.067165 MIRROR_EXTEND to 7 mirrors 000878 1786103630.713474 mapread 0x000b4e thru 0x003d94 (0x3247 bytes) 000880 1786103630.739185 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) 000881 1786103631.095666 mapwrite 0x020a21 thru 0x02697c (0x5f5c bytes) 000884 1786103631.327403 MIRROR_RESYNC 6 mirrors 000886 1786103633.014759 trunc from 0x02697d to 0x03f268 (0x188ec bytes) 000889 1786103633.119876 write 0x00a806 thru 0x013090 (0x888b bytes) 000890 1786103633.148849 mapread 0x003a86 thru 0x0123c8 (0xe943 bytes) 000891 1786103633.206632 read 0x004415 thru 0x012d03 (0xe8ef bytes) 000892 1786103633.213565 trunc from 0x03f269 to 0x02608d (0x191db bytes) 000897 1786103633.259305 write 0x036667 thru 0x03bb34 (0x54ce bytes) 000898 1786103633.270974 MIRROR_RESYNC 6 mirrors 000899 1786103634.449989 trunc from 0x03bb35 to 0x02a66f (0x114c5 bytes) 000900 1786103634.564771 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ba:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a670): Mirror 3: 0x20711997 (stale) Mirror 5: 0x20711997 (stale) Mirror 6: 0x20711997 (stale) Mirror 7: 0x20711997 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a670): Mirror 3: 0x20711997 (stale) Mirror 5: 0x20711997 (stale) Mirror 6: 0x20711997 (stale) Mirror 7: 0x20711997 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a670: skipped 000901 1786103634.956828 write 0x018baf thru 0x026a3f (0xde91 bytes) 000903 1786103634.969900 trunc from 0x02a670 to 0x019d13 (0x1095c bytes) 000905 1786103635.005409 write 0x02b6f7 thru 0x02f324 (0x3c2e bytes) 000906 1786103635.011595 mapread 0x010708 thru 0x017b9e (0x7497 bytes) 000907 1786103635.038626 mapread 0x012002 thru 0x0217ef (0xf7ee bytes) 000910 1786103635.079531 trunc from 0x02f325 to 0x009955 (0x259cf bytes) 000911 1786103635.108990 read 0x007807 thru 0x009955 (0x214f bytes) 000914 1786103635.120301 write 0x03c307 thru 0x03ffff (0x3cf9 bytes) 000915 1786103635.123595 write 0x033db3 thru 0x03ffff (0xc24d bytes) 000917 1786103635.130429 trunc from 0x040000 to 0x0373df (0x8c20 bytes) 000920 1786103635.151842 read 0x019f09 thru 0x027223 (0xd31b bytes) 000921 1786103635.172554 write 0x03a844 thru 0x03ffff (0x57bc bytes) 000922 1786103635.175945 mapwrite 0x0337f9 thru 0x03ffff (0xc807 bytes) 000923 1786103635.299074 trunc from 0x040000 to 0x0396eb (0x6914 bytes) 000924 1786103635.328443 mapwrite 0x00fc70 thru 0x0120be (0x244f bytes) 000925 1786103635.397984 trunc from 0x0396ec to 0x036e95 (0x2856 bytes) 000926 1786103635.423366 mapread 0x0270d2 thru 0x035223 (0xe152 bytes) 000928 1786103635.442126 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36e96): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36e96): Mirror 3: 0x6be2c70 (stale) Mirror 5: 0x6be2c70 (stale) Mirror 6: 0x6be2c70 (stale) Mirror 7: 0x6be2c70 (stale) CRC-32 checksum value for chunk [0, 0x36e96): Mirror 1: 0x49e297db Mirror 8: 0x49e297db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36e96: skipped 000933 1786103636.593329 write 0x036736 thru 0x03ffff (0x98ca bytes) 000934 1786103636.694308 mapread 0x037c47 thru 0x03f933 (0x7ced bytes) 000939 1786103636.703757 trunc from 0x040000 to 0x00c654 (0x339ab bytes) 000940 1786103636.739834 trunc from 0x00c655 to 0x029294 (0x1cc40 bytes) 000941 1786103636.776343 write 0x038640 thru 0x03da32 (0x53f3 bytes) 000942 1786103636.783201 mapwrite 0x012438 thru 0x01d8cf (0xb498 bytes) 000944 1786103636.985482 write 0x02ad92 thru 0x02bfbe (0x122d bytes) 000948 1786103637.022706 mapwrite 0x00af48 thru 0x01264f (0x7708 bytes) 000949 1786103637.091844 mapread 0x00bc3e thru 0x01b76d (0xfb30 bytes) 000950 1786103637.093959 mapwrite 0x034eef thru 0x03fb6b (0xac7d bytes) 000951 1786103637.188326 trunc from 0x03fb6c to 0x0261ca (0x199a1 bytes) 000952 1786103637.219564 trunc from 0x0261cb to 0x01510b (0x110bf bytes) 000953 1786103637.252372 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1510c): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1510c): Mirror 3: 0x4bd2a149 (stale) Mirror 5: 0x4bd2a149 (stale) Mirror 6: 0x4bd2a149 (stale) Mirror 7: 0x4bd2a149 (stale) Mirror 8: 0x5953599a (stale) CRC-32 checksum value for chunk [0, 0x1510c): Mirror 1: 0xba8aa4fe Mirror 9: 0xba8aa4fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1510c: skipped 000955 1786103638.233936 mapwrite 0x021d57 thru 0x02c2b0 (0xa55a bytes) 000956 1786103638.430781 mapwrite 0x037211 thru 0x03ffff (0x8def bytes) 000957 1786103638.555123 read 0x018359 thru 0x02108a (0x8d32 bytes) 000958 1786103638.571982 mapwrite 0x01e49f thru 0x024fdb (0x6b3d bytes) 000959 1786103638.647074 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5b5d4672 (stale) Mirror 5: 0x5b5d4672 (stale) Mirror 6: 0x5b5d4672 (stale) Mirror 7: 0x5b5d4672 (stale) Mirror 8: 0x35bdf3b3 (stale) Mirror 9: 0x177ebd1a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x292eb5cf Mirror 10: 0x292eb5cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000960 1786103639.572691 write 0x003a82 thru 0x007ba7 (0x4126 bytes) 000963 1786103639.698843 trunc from 0x040000 to 0x021388 (0x1ec77 bytes) 000964 1786103639.762523 read 0x00e37d thru 0x00f187 (0xe0b bytes) 000967 1786103639.777390 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x21389): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21389): Mirror 3: 0xa346b069 (stale) Mirror 5: 0xa346b069 (stale) Mirror 6: 0xa346b069 (stale) Mirror 7: 0xa346b069 (stale) Mirror 8: 0xdf81cec1 (stale) Mirror 9: 0xbd9180a9 (stale) Mirror 10: 0xbde2a0d (stale) CRC-32 checksum value for chunk [0, 0x21389): Mirror 1: 0xfcc3b727 Mirror 11: 0xfcc3b727 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21389: skipped 000968 1786103640.984293 mapread 0x01f6c2 thru 0x021388 (0x1cc7 bytes) 000970 1786103641.028298 mapwrite 0x0322e4 thru 0x037c07 (0x5924 bytes) 000971 1786103641.160194 mapread 0x02b2ec thru 0x03635c (0xb071 bytes) 000972 1786103641.166350 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x37c08): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c08): Mirror 3: 0x9233891e (stale) Mirror 5: 0x9233891e (stale) Mirror 6: 0x9233891e (stale) Mirror 7: 0x9233891e (stale) Mirror 8: 0x577493ae (stale) Mirror 9: 0x7fa2943f (stale) Mirror 10: 0x927437bc (stale) Mirror 11: 0x2d3090e6 (stale) CRC-32 checksum value for chunk [0, 0x37c08): Mirror 1: 0x9cf50198 Mirror 12: 0x9cf50198 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c08: skipped 000973 1786103642.207777 read 0x025f74 thru 0x030874 (0xa901 bytes) 000974 1786103642.220213 trunc from 0x037c08 to 0x031d34 (0x5ed3 bytes) 000977 1786103642.353764 write 0x001926 thru 0x00da3d (0xc118 bytes) 000978 1786103642.368133 mapread 0x01add9 thru 0x01d5d2 (0x27fa bytes) 000979 1786103642.387537 read 0x01077d thru 0x018c32 (0x84b6 bytes) 000980 1786103642.401960 mapwrite 0x03ab05 thru 0x03ffff (0x54fb bytes) 000982 1786103642.484757 trunc from 0x040000 to 0x01e906 (0x216f9 bytes) 000986 1786103642.506292 mapread 0x00e71f thru 0x010e26 (0x2708 bytes) 000987 1786103642.507445 trunc from 0x01e907 to 0x023af5 (0x51ef bytes) 000988 1786103642.527796 mapwrite 0x016275 thru 0x0185c4 (0x2350 bytes) 000989 1786103642.609776 mapread 0x001632 thru 0x009b7d (0x854c bytes) 000991 1786103642.620219 read 0x011a7c thru 0x012f83 (0x1508 bytes) 000992 1786103642.621055 write 0x003da1 thru 0x009030 (0x5290 bytes) 000993 1786103642.623503 mapread 0x0093e9 thru 0x016d83 (0xd99b bytes) 000995 1786103642.626279 mapread 0x00956a thru 0x00bdbf (0x2856 bytes) 000996 1786103642.628387 trunc from 0x023af6 to 0x01c1a1 (0x7954 bytes) 000999 1786103642.666021 trunc from 0x01c1a2 to 0x02842b (0xc28a bytes) 001000 1786103642.683609 mapwrite 0x01e131 thru 0x022a0c (0x48dc bytes) All operations completed A-OK! PASS 70a (80s) == sanity-flr test 71: check mirror extend parent fid ==== 07:54:08 (1786103648) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 07:54:11 (1786103651) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.262881 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16021 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 07:54:20 (1786103660) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6279648 .. Extending file size to 5628832 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8611520 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7443392 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7489120 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8115808 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8215520 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8550176 .. Extending file size to 8869376 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9176384 .. Waiting 57473 57474 57475 57476 57477 done 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) 192.168.206.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:55:42 (1786103742) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1577440 .. Extending file size to 1444096 .. failed Extending file size to 1626112 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6818944 .. Extending file size to 8510752 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7809856 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8156832 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8317728 .. failed Extending file size to 8408608 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8617056 .. Extending file size to 8706560 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8711104 .. done Extending file size to 9033632 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9261920 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58985 58986 58989 58990 58992 failed 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) 192.168.206.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:57:00 (1786103820) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117251 s, 17.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.289596 s, 7.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3fd3d998 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:57:15 (1786103835) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0877394 s, 23.9 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 07:57:21 (1786103841) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x204:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:57:26 (1786103846) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116813 s, 18.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:57:32 (1786103852) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135051 s, 15.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:57:38 (1786103858) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165729 s, 12.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:58:08 (1786103888) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg637-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112533 s, 9.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:59:02 (1786103942) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg637-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169823 s, 12.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:59:56 (1786103996) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg637-server mds-ost sync done. Creating new pool test_204f oleg637-server: Pool lustre.test_204f created Adding targets to pool oleg637-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg637-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16353 s, 12.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg637-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg637-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg637-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:01:07 (1786104067) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:01:12 (1786104072) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:01:17 (1786104077) oleg637-server: Pool lustre.test_206 created Adding targets to pool oleg637-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg637-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg637-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg637-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg637-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:01:38 (1786104098) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937975 s, 11.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827707 s, 12.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7e4c4e80 Mirror 3: 0x7e4c4e80 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:01:47 (1786104107) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:01:50 (1786104110) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:01:52 (1786104112) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.114736 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1cc0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ae0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ae0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ae0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1cc0000 Read this (4 bytes): BBBB 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:02:20 (1786104140) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.010831 s, 756 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:02:25 (1786104145) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388868 s, 27.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x229 lmm_fid: [0x200000404:0x229:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x229 lmm_fid: [0x200000404:0x229:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834473 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-07 08:02:26.000000000 -0400 Modify: 2026-08-07 08:02:26.000000000 -0400 Change: 2026-08-07 08:02:27.000000000 -0400 Birth: 2026-08-07 08:02:26.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:02:31 (1786104151) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:02:34 (1786104154) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.455322 s, 90.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0443602 s, 92.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 7 08:02 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 7 08:02 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:02:42 (1786104162) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:02:49 (1786104169) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1580 sec =========== 08:02:57 (1786104177) === sanity-flr: start cleanup 08:02:59 (1786104179) === === sanity-flr: finish cleanup 08:03:05 (1786104185) ===