-----============= acceptance-small: sanity-flr ============----- Tue Sep 15 20:24:47 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg631-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 20:24:58 (1789518298) === oleg631-client.virtnet: executing check_config_client /mnt/lustre oleg631-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg631-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff89e784e56000.idle_timeout=debug osc.lustre-OST0001-osc-ffff89e784e56000.idle_timeout=debug disable quota as required oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:25:17 (1789518317) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8207] == sanity-flr test 0a: lfs mirror create with -N option == 20:25:18 (1789518318) PASS 0a (8s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:25:26 (1789518326) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:25:28 (1789518328) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:25:30 (1789518330) PASS 0d (11s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:25:41 (1789518341) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:25:43 (1789518343) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 20:25:45 (1789518345) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff89e784e56000.stats=clear osc.lustre-OST0001-osc-ffff89e784e56000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:25:49 (1789518349) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:25:54 (1789518354) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:25:59 (1789518359) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:26:01 (1789518361) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:26:05 (1789518365) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 20:26:10 (1789518370) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:26:13 (1789518373) PASS 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:26:18 (1789518378) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:26:22 (1789518382) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:26:27 (1789518387) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:26:31 (1789518391) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.177556 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035365 s, 29.7 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:26:34 (1789518394) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0665419 s, 78.8 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:26:38 (1789518398) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:26:43 (1789518403) Stopping /mnt/lustre-ost1 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89e784e56000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89e784e56000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg631-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89e784e56000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89e784e56000.ost_server_uuid in DISCONN state after 1 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg631-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (39s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:27:22 (1789518442) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.294118 s, 32.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89e784e56000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89e784e56000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg631-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (23s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:27:45 (1789518465) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89e784e56000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89e784e56000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg631-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89e784e56000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89e784e56000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg631-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (44s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:28:29 (1789518509) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35514 s, 38.7 MB/s -rw-r--r-- 1 root root 52428800 Sep 15 20:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.622s user 0m0.000s sys 0m0.181s Stopping /mnt/lustre-ost1 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89e784e56000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89e784e56000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.618s user 0m0.000s sys 0m0.182s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg631-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg631-server oleg631-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89e784e56000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89e784e56000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.568s user 0m0.000s sys 0m0.173s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg631-server: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg631-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg631-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89e784e56000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89e784e56000.ost_server_uuid in FULL state after 0 sec PASS 33b (49s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:29:18 (1789518558) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:29:20 (1789518560) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:29:22 (1789518562) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 20:29:24 (1789518564) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 20:29:28 (1789518568) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.255972 s, 49.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (5s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:29:33 (1789518573) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173012 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174599 s, 60.1 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:29:37 (1789518577) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:29:42 (1789518582) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00106519 s, 961 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00399597 s, 256 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000682583 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114102 s, 897 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 20:29:47 (1789518587) create test files with size 15760871 .. adc33b81822b0dd77bd62c02584d49ce - e80ae1df350249a8b04b2e0e4bfbab3a - c7fde17554a5c15bf5e7786c73fffd40 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg631-client.virtnet /mnt/lustre (opts:) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 20:29:58 (1789518598) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:30:05 (1789518605) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:30:09 (1789518609) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545146 s, 19.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245148 s, 42.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb1:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb2:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 20:30:13 (1789518613) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0483195 s, 43.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.16239 s, 25.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115 s, 18.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (8s) == sanity-flr test 42: lfs mirror verify ================= 20:30:21 (1789518621) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 20:30:23 (1789518623) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:30:25 (1789518625) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185439 s, 56.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 20:30:29 (1789518629) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.168783 s, 18.6 MB/s PASS 44a (6s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:30:35 (1789518635) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:30:40 (1789518640) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.144524 s, 72.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg631-client.virtnet /mnt/lustre (opts:) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:30:49 (1789518649) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff89e787107000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.157666 s, 66.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg631-client.virtnet /mnt/lustre (opts:) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff89e7a0ea3000.xattr_cache=1 PASS 44e (8s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:30:57 (1789518657) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (23s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:31:20 (1789518680) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (20s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:31:40 (1789518700) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 20:31:42 (1789518702) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.135405 s, 23.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0930236 s, 33.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x19b663a7 (nosync,stale) Mirror 2: 0x19b663a7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: c78778d21da942d9b9db2228b61d94e3 - mirror 1 checksum: 5d64363d61e4af0b775fe2130dbce712 - mirror 2 checksum: c78778d21da942d9b9db2228b61d94e3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-15T20:31:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-15T20:31:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c78778d21da942d9b9db2228b61d94e3 - mirror 1 checksum: 5d64363d61e4af0b775fe2130dbce712 - mirror 2 checksum: 5d64363d61e4af0b775fe2130dbce712 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-15T20:31:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-15T20:31:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:31:47 (1789518707) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179306 s, 58.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (4s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:31:51 (1789518711) Failing mds1 on oleg631-server Stopping /mnt/lustre-mds1 (opts:) on oleg631-server 20:32:00 (1789518720) shut down facet: mds1 facet_host: oleg631-server facet_failover_host: oleg631-server Failover mds1 to oleg631-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 20:32:17 (1789518737) targets are mounted 20:32:17 (1789518737) facet_failover done oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203171 s, 51.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } PASS 50A (36s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:32:28 (1789518748) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0272102 s, 753 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0202649 s, 1.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:32:33 (1789518753) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0318747 s, 643 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00889783 s, 2.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (6s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:32:39 (1789518759) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00520192 s, 3.1 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:32:43 (1789518763) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0262201 s, 781 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:32:47 (1789518767) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00592195 s, 692 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0266497 s, 154 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:32:51 (1789518771) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (13s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:33:04 (1789518784) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (22s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:33:27 (1789518807) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:33:47 (1789518827) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (20s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:34:07 (1789518847) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 20:34:11 (1789518851) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57478 mirror split pid 57479 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57478 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57479 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 20:35:15 (1789518915) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 8916 skipping zero length zero range truncating to largest ever: 0xabb 000003 1789518916.854056 trunc from 00000000 to 0x000aba (0xabb bytes) 000005 1789518916.871887 mapwrite 0x019053 thru 0x024723 (0xb6d1 bytes) 000006 1789518916.955059 read 0x00fba8 thru 0x012d64 (0x31bd bytes) 000007 1789518916.963483 mapwrite 0x0257e9 thru 0x02cd7a (0x7592 bytes) 000008 1789518917.025495 mapwrite 0x01ab55 thru 0x01c701 (0x1bad bytes) 000009 1789518917.050262 mapwrite 0x02ce16 thru 0x0330fe (0x62e9 bytes) truncating to largest ever: 0xfa3c 000010 1789518917.105600 trunc from 0x0330ff to 0x00fa3b (0x236c3 bytes) 000012 1789518917.122562 read 0x00dba4 thru 0x00fa3b (0x1e98 bytes) 000013 1789518917.145699 MIRROR_EXTEND to 2 mirrors 000014 1789518917.617946 MIRROR_RESYNC 2 mirrors 000015 1789518917.839408 write 0x0050a8 thru 0x00b8f0 (0x6849 bytes) 000016 1789518917.873454 punch from 0x00976a to 0x00fa3b (0x62d2 bytes) fallocating to largest ever: 0x1035d 000017 1789518917.885186 fallocate 0x00557b thru 0x01035c (0xade2 bytes) 000018 1789518917.888855 mapwrite 0x028611 thru 0x0303de (0x7dce bytes) 000019 1789518917.959858 write 0x02f515 thru 0x03e98b (0xf477 bytes) 000021 1789518917.962324 mapwrite 0x024353 thru 0x02b46d (0x711b bytes) 000022 1789518917.994753 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) fallocating to largest ever: 0x40000 000023 1789518918.099740 fallocate 0x03e0b0 thru 0x03ffff (0x1f50 bytes) 000024 1789518918.104281 read 0x02ca12 thru 0x036670 (0x9c5f bytes) 000025 1789518918.112683 fallocate 0x039827 thru 0x03ffff (0x67d9 bytes) 000026 1789518918.116312 read 0x0133c7 thru 0x01f16d (0xbda7 bytes) 000027 1789518918.126529 read 0x00c53f thru 0x018e53 (0xc915 bytes) 000028 1789518918.127183 trunc from 0x03e98c to 0x00a4c0 (0x344cb bytes) 000029 1789518918.143506 write 0x0308e7 thru 0x03d686 (0xcda0 bytes) 000030 1789518918.144942 trunc from 0x02c33e to 0x03258f (0x6252 bytes) 000031 1789518918.164423 write 0x00a4b8 thru 0x00d66c (0x31b5 bytes) 000033 1789518918.172949 read 0x00d675 thru 0x019c12 (0xc59e bytes) 000034 1789518918.187963 mapread 0x036d8e thru 0x03b9a9 (0x4c1c bytes) 000035 1789518918.190543 write 0x0298ac thru 0x02b94a (0x209f bytes) 000036 1789518918.194341 fallocate 0x01a5f8 thru 0x023318 (0x8d21 bytes) 000037 1789518918.199614 write 0x02a85a thru 0x03142b (0x6bd2 bytes) 000038 1789518918.204007 trunc from 0x01e693 to 0x01f1c3 (0xb31 bytes) 000039 1789518918.209466 mapwrite 0x017f1a thru 0x01ed67 (0x6e4e bytes) 000040 1789518918.236194 trunc from 0x00d6a8 to 0x01497e (0x72d7 bytes) 000041 1789518918.244007 mapread 0x0226b5 thru 0x02abe3 (0x852f bytes) 000042 1789518918.245374 mapread 0x02a36e thru 0x036659 (0xc2ec bytes) 000043 1789518918.246859 trunc from 0x0302c7 to 0x03625a (0x5f94 bytes) 000044 1789518918.251975 fallocate 0x02d3aa thru 0x03bd3d (0xe994 bytes) 000045 1789518918.257321 trunc from 0x0008f2 to 0x00fa1b (0xf12a bytes) 000046 1789518918.263509 mapwrite 0x02fda6 thru 0x038ebb (0x9116 bytes) 000047 1789518918.310213 mapread 0x03706b thru 0x03d686 (0x661c bytes) 000048 1789518918.311141 fallocate 0x03e972 thru 0x03ffff (0x168e bytes) 000049 1789518918.317518 fallocate 0x03a1d5 thru 0x03ffff (0x5e2b bytes) 000050 1789518918.322529 trunc from 0x03d687 to 0x013252 (0x2a434 bytes) 000051 1789518918.337357 mapread 0x011289 thru 0x013252 (0x1fca bytes) 000052 1789518918.353472 read 0x0064f9 thru 0x00f31b (0x8e23 bytes) 000053 1789518918.362806 fallocate 0x0027bb thru 0x00c32c (0x9b72 bytes) 000054 1789518918.375577 MIRROR_EXTEND to 2 mirrors 000055 1789518918.800840 trunc from 0x001876 to 0x005ba9 (0x4334 bytes) 000056 1789518918.839233 mapwrite 0x00a53f thru 0x00ce53 (0x2915 bytes) 000057 1789518918.874080 read 0x00d2e9 thru 0x013252 (0x5f6a bytes) 000058 1789518918.875471 write 0x029164 thru 0x035e9e (0xcd3b bytes) 000059 1789518918.886225 MIRROR_RESYNC 2 mirrors 000060 1789518919.414788 trunc from 0x01de8f to 0x01fcc0 (0x1e32 bytes) 000061 1789518919.445358 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) 000062 1789518919.560168 punch from 0x000b30 to 0x00ce88 (0xc359 bytes) 000063 1789518919.564403 punch from 0x02a30c to 0x035e9e (0xbb93 bytes) 000064 1789518919.568732 mapread 0x023239 thru 0x02db16 (0xa8de bytes) 000065 1789518919.590990 read 0x007d7d thru 0x007e87 (0x10b bytes) 000066 1789518919.597839 mapread 0x032c35 thru 0x035e9e (0x326a bytes) 000067 1789518919.606073 write 0x0362de thru 0x03825f (0x1f82 bytes) 000068 1789518919.611328 mapread 0x02803a thru 0x034d74 (0xcd3b bytes) 000069 1789518919.612973 trunc from 0x038260 to 0x00b2ba (0x2cfa5 bytes) 000070 1789518919.628627 write 0x01cb8d thru 0x027e97 (0xb30b bytes) 000072 1789518919.632633 punch from 0x015e25 to 0x025a91 (0xfc6d bytes) 000074 1789518919.644963 mapwrite 0x016ff5 thru 0x01878a (0x1796 bytes) 000075 1789518919.665954 read 0x020805 thru 0x0242e9 (0x3ae5 bytes) 000078 1789518919.674843 write 0x02db15 thru 0x0355e3 (0x7acf bytes) 000079 1789518919.676535 mapread 0x0095b9 thru 0x012cc8 (0x9710 bytes) 000080 1789518919.677659 trunc from 0x0355e4 to 0x01062f (0x24fb4 bytes) 000081 1789518919.689862 trunc from 0x00a76c to 0x010584 (0x5e19 bytes) 000082 1789518919.694749 trunc from 0x003bb9 to 0x008e7e (0x52c6 bytes) 000085 1789518919.700911 mapread 0x001bd4 thru 0x01062f (0xea5c bytes) 000086 1789518919.744043 write 0x00be78 thru 0x0168b7 (0xaa40 bytes) 000087 1789518919.746605 trunc from 0x0168b8 to 0x021bc7 (0xb310 bytes) 000088 1789518919.764363 read 0x010493 thru 0x018a73 (0x85e1 bytes) 000089 1789518919.772897 read 0x00c5ef thru 0x01ba43 (0xf455 bytes) 000090 1789518919.773516 mapread 0x004d55 thru 0x00daeb (0x8d97 bytes) 000091 1789518919.775077 mapread 0x00094f thru 0x004a00 (0x40b2 bytes) 000093 1789518919.776870 write 0x009732 thru 0x015d48 (0xc617 bytes) 000094 1789518919.779315 trunc from 0x0115d1 to 0x012564 (0xf94 bytes) 000095 1789518919.800649 mapwrite 0x0076dd thru 0x00ffde (0x8902 bytes) 000096 1789518919.820462 trunc from 0x021bc8 to 0x0325b6 (0x109ef bytes) 000097 1789518919.833219 write 0x018427 thru 0x02733b (0xef15 bytes) 000098 1789518919.848067 fallocate 0x008081 thru 0x011c6a (0x9bea bytes) 000099 1789518919.856439 trunc from 0x0325b7 to 0x03e16b (0xbbb5 bytes) 000100 1789518919.869700 trunc from 0x0289b5 to 0x032c94 (0xa2e0 bytes) 000102 1789518919.875673 trunc from 0x019763 to 0x01b895 (0x2133 bytes) 000103 1789518919.897045 punch from 0x00e858 to 0x01cdfc (0xe5a5 bytes) 000106 1789518919.910231 mapread 0x018b16 thru 0x01df37 (0x5422 bytes) 000107 1789518919.922810 punch from 0x00ba32 to 0x019967 (0xdf36 bytes) 000108 1789518919.927591 punch from 0x00fa06 to 0x015d7a (0x6375 bytes) 000110 1789518919.935651 trunc from 0x012c08 to 0x013906 (0xcff bytes) 000111 1789518919.940876 read 0x028c95 thru 0x0348d0 (0xbc3c bytes) 000112 1789518919.941634 trunc from 0x037dae to 0x03db6f (0x5dc2 bytes) 000113 1789518919.946568 punch from 0x01ca33 to 0x02a2f5 (0xd8c3 bytes) 000114 1789518919.955495 mapread 0x03523b thru 0x038d57 (0x3b1d bytes) 000115 1789518919.967675 read 0x00a5ba thru 0x014f49 (0xa990 bytes) 000117 1789518919.968856 mapread 0x017925 thru 0x0198b2 (0x1f8e bytes) 000118 1789518919.970163 punch from 0x03cade to 0x03e16b (0x168e bytes) 000119 1789518919.978045 mapwrite 0x02c7e6 thru 0x0395bd (0xcdd8 bytes) 000120 1789518920.008825 punch from 0x01681a to 0x022dde (0xc5c5 bytes) 000121 1789518920.014523 read 0x0292c5 thru 0x0348d7 (0xb613 bytes) 000122 1789518920.015840 mapread 0x0141bf thru 0x0210a3 (0xcee5 bytes) 000124 1789518920.032533 write 0x01abe7 thru 0x01d405 (0x281f bytes) 000125 1789518920.033974 trunc from 0x02f24f to 0x0340fd (0x4eaf bytes) 000126 1789518920.038509 trunc from 0x03e16c to 0x035495 (0x8cd6 bytes) 000128 1789518920.053638 fallocate 0x0036fd thru 0x0099f4 (0x62f8 bytes) 000129 1789518920.058456 write 0x0139a4 thru 0x01a511 (0x6b6e bytes) 000130 1789518920.060341 fallocate 0x0279c5 thru 0x02f1b1 (0x77ed bytes) 000131 1789518920.074470 trunc from 0x033388 to 0x035495 (0x210e bytes) 000132 1789518920.079300 read 0x00ada9 thru 0x019837 (0xea8f bytes) 000133 1789518920.080015 read 0x0131b9 thru 0x01a336 (0x717e bytes) 000135 1789518920.081085 write 0x0071c3 thru 0x00821c (0x105a bytes) 000137 1789518920.083247 trunc from 0x019980 to 0x01b4ab (0x1b2c bytes) 000139 1789518920.095319 trunc from 0x031525 to 0x03230e (0xdea bytes) 000141 1789518920.100945 mapwrite 0x016d25 thru 0x021ed7 (0xb1b3 bytes) 000142 1789518920.126493 trunc from 0x014e0e to 0x022681 (0xd874 bytes) 000143 1789518920.131920 write 0x028f5c thru 0x02c678 (0x371d bytes) 000144 1789518920.133435 fallocate 0x037a28 thru 0x0387e6 (0xdbf bytes) 000145 1789518920.138554 fallocate 0x02d65a thru 0x033ad1 (0x6478 bytes) 000146 1789518920.142424 fallocate 0x007bf9 thru 0x00dd66 (0x616e bytes) 000148 1789518920.146941 punch from 0x00639b to 0x01219e (0xbe04 bytes) 000149 1789518920.153658 write 0x01b2e5 thru 0x01d376 (0x2092 bytes) 000150 1789518920.167268 mapwrite 0x012712 thru 0x01b3e2 (0x8cd1 bytes) 000151 1789518920.230422 mapread 0x0173a5 thru 0x02162c (0xa288 bytes) 000152 1789518920.252522 mapread 0x028491 thru 0x031b49 (0x96b9 bytes) 000153 1789518920.256091 trunc from 0x0114db to 0x015467 (0x3f8d bytes) 000154 1789518920.261365 mapwrite 0x0077bb thru 0x00da1b (0x6261 bytes) 000155 1789518920.329214 MIRROR_EXTEND to 2 mirrors 000156 1789518920.875223 mapwrite 0x039048 thru 0x03ffff (0x6fb8 bytes) 000157 1789518920.991142 trunc from 0x040000 to 0x0369c7 (0x9638 bytes) 000159 1789518921.020420 mapwrite 0x02deb1 thru 0x03b6b1 (0xd801 bytes) 000160 1789518921.082395 punch from 0x00f92e to 0x014be8 (0x52bb bytes) 000161 1789518921.096227 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b6b2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6b2): Mirror 2: 0x5bcf0a73 (stale) CRC-32 checksum value for chunk [0, 0x3b6b2): Mirror 1: 0xdc83157d Mirror 3: 0xdc83157d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6b2: skipped 000162 1789518921.757514 write 0x00ad7e thru 0x00d86b (0x2aee bytes) 000163 1789518921.797480 trunc from 0x03b6b2 to 0x02e027 (0xd68a bytes) 000164 1789518921.827040 read 0x02a5f6 thru 0x02cfd0 (0x29db bytes) 000165 1789518921.834769 mapwrite 0x03301c thru 0x03a1de (0x71c3 bytes) 000166 1789518921.905278 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a1df): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1df): Mirror 2: 0xc1570f14 (stale) Mirror 3: 0xe1a79119 (stale) CRC-32 checksum value for chunk [0, 0x3a1df): Mirror 1: 0x355c3fbf Mirror 4: 0x355c3fbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1df: skipped 000167 1789518922.598864 mapwrite 0x00b979 thru 0x00fd68 (0x43f0 bytes) 000168 1789518922.678988 write 0x00449f thru 0x00c927 (0x8489 bytes) 000169 1789518922.689512 fallocate 0x0184e3 thru 0x027885 (0xf3a3 bytes) 000170 1789518922.696007 trunc from 0x033fd9 to 0x03a1de (0x6206 bytes) 000171 1789518922.700438 write 0x036b4a thru 0x03ffff (0x94b6 bytes) 000172 1789518922.723395 MIRROR_RESYNC 4 mirrors 000173 1789518923.913258 mapwrite 0x0109ec thru 0x01613d (0x5752 bytes) 000174 1789518923.979614 read 0x011988 thru 0x01fffd (0xe676 bytes) 000175 1789518923.993288 punch from 0x029248 to 0x02b926 (0x26df bytes) 000176 1789518923.998202 punch from 0x027591 to 0x034cf0 (0xd760 bytes) 000177 1789518924.004061 mapread 0x0192c2 thru 0x01ccb1 (0x39f0 bytes) 000178 1789518924.005729 write 0x0221d8 thru 0x0246d2 (0x24fb bytes) 000179 1789518924.021436 write 0x033bcd thru 0x038545 (0x4979 bytes) 000180 1789518924.031343 mapwrite 0x032329 thru 0x03f666 (0xd33e bytes) 000181 1789518924.073480 trunc from 0x015803 to 0x01de82 (0x8680 bytes) 000182 1789518924.085375 MIRROR_RESYNC 4 mirrors 000183 1789518925.010428 trunc from 0x01ec35 to 0x02635f (0x772b bytes) 000184 1789518925.051931 read 0x0059ae thru 0x00fc77 (0xa2ca bytes) 000185 1789518925.060148 mapwrite 0x00e00e thru 0x0131f2 (0x51e5 bytes) 000186 1789518925.093349 fallocate 0x017e9a thru 0x0249c6 (0xcb2d bytes) 000187 1789518925.097286 trunc from 0x00dd6b to 0x01c686 (0xe91c bytes) 000188 1789518925.108542 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x856e32e6 (stale) Mirror 3: 0x856e32e6 (stale) Mirror 4: 0x856e32e6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa800b65c Mirror 5: 0xa800b65c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000189 1789518925.809481 trunc from 0x040000 to 0x0120d6 (0x2df29 bytes) 000190 1789518925.851127 mapread 0x00fb1f thru 0x0120d6 (0x25b8 bytes) 000191 1789518925.875893 write 0x02ac84 thru 0x033c62 (0x8fdf bytes) 000192 1789518925.877503 punch from 0x00411a to 0x00eea7 (0xad8e bytes) 000193 1789518925.882939 mapread 0x00c52d thru 0x012ea1 (0x6975 bytes) 000194 1789518925.895667 trunc from 0x033c63 to 0x03e6f4 (0xaa92 bytes) 000195 1789518925.914039 MIRROR_RESYNC 5 mirrors 000196 1789518926.929759 MIRROR_EXTEND to 6 mirrors 000197 1789518927.475633 mapread 0x02d834 thru 0x02e9da (0x11a7 bytes) 000198 1789518927.483282 fallocate 0x019fef thru 0x02189d (0x78af bytes) 000199 1789518927.506408 write 0x007fc1 thru 0x01655c (0xe59c bytes) 000201 1789518927.521925 mapwrite 0x03c137 thru 0x03ffff (0x3ec9 bytes) 000202 1789518927.562721 trunc from 0x040000 to 0x001983 (0x3e67c bytes) 000203 1789518927.574466 mapwrite 0x00f771 thru 0x017341 (0x7bd1 bytes) 000204 1789518927.628598 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17342): Mirror 3: 0x11247b98 (stale) Mirror 4: 0x11247b98 (stale) Mirror 5: 0x11247b98 (stale) Mirror 6: 0x11247b98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17342): Mirror 3: 0x11247b98 (stale) Mirror 4: 0x11247b98 (stale) Mirror 5: 0x11247b98 (stale) Mirror 6: 0x11247b98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17342: skipped 000205 1789518927.994353 mapread 0x0147cc thru 0x017341 (0x2b76 bytes) 000206 1789518928.009861 trunc from 0x017342 to 0x0160cd (0x1274 bytes) 000207 1789518928.025150 write 0x0244f1 thru 0x032078 (0xdb88 bytes) 000209 1789518928.027687 punch from 0x01947b to 0x023c56 (0xa7dc bytes) 000211 1789518928.033318 write 0x03ae13 thru 0x03ffff (0x51ed bytes) 000212 1789518928.035072 mapread 0x00624e thru 0x01212d (0xbee0 bytes) 000213 1789518928.048588 trunc from 0x0038b4 to 0x0103d1 (0xcb1e bytes) 000214 1789518928.055058 mapwrite 0x0231e3 thru 0x02c5a4 (0x93c2 bytes) 000215 1789518928.084923 fallocate 0x00beb2 thru 0x01030b (0x445a bytes) 000216 1789518928.090585 trunc from 0x040000 to 0x004e64 (0x3b19b bytes) 000217 1789518928.103917 read 0x003c70 thru 0x004e64 (0x11f5 bytes) 000218 1789518928.108492 read 0x004e4d thru 0x004e64 (0x18 bytes) 000219 1789518928.109086 write 0x0352b3 thru 0x036362 (0x10b0 bytes) 000220 1789518928.110187 read 0x002098 thru 0x00c5e5 (0xa54e bytes) 000221 1789518928.123154 mapread 0x00cd0e thru 0x01bfe5 (0xf2d8 bytes) 000222 1789518928.127596 MIRROR_RESYNC 5 mirrors 000223 1789518928.700613 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) 000224 1789518928.909945 punch from 0x0241d9 to 0x02a5b8 (0x63e0 bytes) 000225 1789518928.943197 MIRROR_RESYNC 4 mirrors 000226 1789518929.436139 trunc from 0x036363 to 0x033544 (0x2e1e bytes) 000227 1789518929.461062 mapwrite 0x007dc8 thru 0x00fd15 (0x7f4e bytes) 000228 1789518929.508001 punch from 0x023f6a to 0x031e76 (0xdf0d bytes) 000229 1789518929.514340 punch from 0x022c3d to 0x02a038 (0x73fc bytes) 000231 1789518929.521169 fallocate 0x02729e thru 0x02acb1 (0x3a14 bytes) 000232 1789518929.525671 read 0x017f49 thru 0x01af14 (0x2fcc bytes) 000233 1789518929.531030 write 0x01d87a thru 0x021a23 (0x41aa bytes) 000234 1789518929.541582 trunc from 0x033545 to 0x03ff76 (0xca32 bytes) 000235 1789518929.555485 mapread 0x02f89d thru 0x03c416 (0xcb7a bytes) 000236 1789518929.571438 trunc from 0x015de3 to 0x017601 (0x181f bytes) 000238 1789518929.577968 mapwrite 0x02a2b9 thru 0x0354e0 (0xb228 bytes) 000239 1789518929.599525 mapwrite 0x039d11 thru 0x03ffff (0x62ef bytes) 000240 1789518929.637926 mapread 0x01dc64 thru 0x01f8a0 (0x1c3d bytes) 000241 1789518929.642611 trunc from 0x01a3d3 to 0x021edc (0x7b0a bytes) 000242 1789518929.651050 trunc from 0x040000 to 0x031339 (0xecc6 bytes) 000243 1789518929.686312 write 0x02c3e7 thru 0x035dba (0x99d4 bytes) 000244 1789518929.687844 punch from 0x0343b0 to 0x035dba (0x1a0b bytes) 000245 1789518929.704685 trunc from 0x032266 to 0x035dba (0x3b55 bytes) 000246 1789518929.709550 trunc from 0x00e063 to 0x01d0f5 (0xf093 bytes) 000247 1789518929.716420 mapread 0x02cf36 thru 0x035dba (0x8e85 bytes) 000248 1789518929.723292 read 0x01f37c thru 0x02e49c (0xf121 bytes) 000249 1789518929.736291 trunc from 0x00a024 to 0x00c2b9 (0x2296 bytes) 000250 1789518929.740213 write 0x01f925 thru 0x02f5a2 (0xfc7e bytes) 000251 1789518929.741879 mapwrite 0x006930 thru 0x00b8a0 (0x4f71 bytes) 000252 1789518929.767253 trunc from 0x035dbb to 0x01fb01 (0x162b9 bytes) 000253 1789518929.782518 fallocate 0x01d496 thru 0x0207c3 (0x332e bytes) 000254 1789518929.787986 trunc from 0x00a735 to 0x0193f0 (0xecbc bytes) 000255 1789518929.793578 punch from 0x001208 to 0x00ac8b (0x9a84 bytes) 000256 1789518929.801802 write 0x0031eb thru 0x00cf4c (0x9d62 bytes) 000257 1789518929.814078 read 0x00640f thru 0x014da3 (0xe995 bytes) 000258 1789518929.822524 write 0x024f4b thru 0x030eba (0xbf70 bytes) 000259 1789518929.824680 read 0x000624 thru 0x007795 (0x7172 bytes) 000260 1789518929.826344 fallocate 0x010a58 thru 0x0154c8 (0x4a71 bytes) 000261 1789518929.830986 mapread 0x01f3e6 thru 0x02ca1f (0xd63a bytes) 000262 1789518929.837056 mapwrite 0x018ddf thru 0x0225d3 (0x97f5 bytes) 000263 1789518929.864341 trunc from 0x012d6e to 0x01b09b (0x832e bytes) 000264 1789518929.871859 read 0x002f6b thru 0x011b3d (0xebd3 bytes) 000265 1789518929.875080 write 0x02fc94 thru 0x0361f8 (0x6565 bytes) 000266 1789518929.891087 MIRROR_RESYNC 4 mirrors 000267 1789518930.724680 fallocate 0x00f0b9 thru 0x013d09 (0x4c51 bytes) 000268 1789518930.758756 trunc from 0x0361f9 to 0x03ec82 (0x8a8a bytes) 000269 1789518930.776190 trunc from 0x03ec83 to 0x024913 (0x1a36f bytes) 000270 1789518930.794517 trunc from 0x024914 to 0x027088 (0x2775 bytes) 000271 1789518930.809864 trunc from 0x027089 to 0x021b94 (0x54f4 bytes) 000272 1789518930.828749 trunc from 0x021b95 to 0x028ce2 (0x714e bytes) 000273 1789518930.839375 trunc from 0x012345 to 0x01d4bb (0xb177 bytes) 000274 1789518930.849818 trunc from 0x028ce3 to 0x015a09 (0x132d9 bytes) 000275 1789518930.864621 write 0x025638 thru 0x0298f7 (0x42c0 bytes) 000276 1789518930.865944 mapwrite 0x035342 thru 0x03ffff (0xacbe bytes) 000277 1789518930.909610 trunc from 0x02e186 to 0x03b88c (0xd707 bytes) 000278 1789518930.919882 punch from 0x03a7c6 to 0x03ca68 (0x22a3 bytes) 000279 1789518930.925692 write 0x0268bc thru 0x030998 (0xa0dd bytes) 000280 1789518930.940361 MIRROR_RESYNC 4 mirrors 000282 1789518931.854913 read 0x03cdeb thru 0x03ffff (0x3215 bytes) 000283 1789518931.860010 mapread 0x01cb83 thru 0x023ca3 (0x7121 bytes) 000285 1789518931.876568 fallocate 0x004822 thru 0x00eaba (0xa299 bytes) 000286 1789518931.906285 fallocate 0x02b155 thru 0x02dafe (0x29aa bytes) 000287 1789518931.914326 trunc from 0x01e875 to 0x02cba2 (0xe32e bytes) 000289 1789518931.926526 fallocate 0x03b049 thru 0x03d260 (0x2218 bytes) 000290 1789518931.933608 read 0x014b95 thru 0x020784 (0xbbf0 bytes) 000291 1789518931.949902 trunc from 0x040000 to 0x0159e9 (0x2a616 bytes) 000292 1789518931.978438 mapwrite 0x01f7b8 thru 0x02da60 (0xe2a9 bytes) 000293 1789518932.078947 trunc from 0x02da61 to 0x0335a1 (0x5b41 bytes) 000294 1789518932.097695 mapread 0x00edb0 thru 0x01b3a0 (0xc5f1 bytes) 000295 1789518932.142354 read 0x00af65 thru 0x012627 (0x76c3 bytes) 000296 1789518932.142999 fallocate 0x024a01 thru 0x02da41 (0x9041 bytes) 000297 1789518932.147393 fallocate 0x03f5e4 thru 0x03ffff (0xa1c bytes) 000298 1789518932.151851 read 0x0120fb thru 0x017c75 (0x5b7b bytes) 000299 1789518932.152477 punch from 0x018cdf to 0x019b56 (0xe78 bytes) 000300 1789518932.167154 trunc from 0x0335a2 to 0x00cc25 (0x2697c bytes) 000301 1789518932.186972 trunc from 0x00cc26 to 0x03c16b (0x2f546 bytes) 000302 1789518932.204792 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c16c): Mirror 4: 0xa34c9c76 (stale) Mirror 6: 0xa34c9c76 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c16c): Mirror 4: 0xa34c9c76 (stale) Mirror 6: 0xa34c9c76 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c16c: skipped 000303 1789518932.487914 write 0x01b025 thru 0x02a7a7 (0xf783 bytes) 000304 1789518932.508296 fallocate 0x01c2d4 thru 0x0261ed (0x9f1a bytes) 000305 1789518932.513349 punch from 0x02415f to 0x0259d8 (0x187a bytes) 000306 1789518932.529221 punch from 0x0248dd to 0x02d64c (0x8d70 bytes) 000307 1789518932.534191 write 0x03013c thru 0x0368c7 (0x678c bytes) 000308 1789518932.551914 fallocate 0x0092dd thru 0x00e938 (0x565c bytes) 000309 1789518932.559294 mapwrite 0x028c87 thru 0x03064c (0x79c6 bytes) 000310 1789518932.606762 read 0x01b26c thru 0x01fd30 (0x4ac5 bytes) 000311 1789518932.607575 write 0x004578 thru 0x005cac (0x1735 bytes) 000312 1789518932.616385 trunc from 0x03c16c to 0x00e1a0 (0x2dfcb bytes) 000313 1789518932.629892 mapwrite 0x03e33a thru 0x03ffff (0x1cc6 bytes) 000314 1789518932.664739 mapread 0x038737 thru 0x03f84e (0x7118 bytes) 000315 1789518932.665722 fallocate 0x01a164 thru 0x020d8e (0x6c2b bytes) 000316 1789518932.670587 punch from 0x03699b to 0x037b15 (0x117b bytes) 000317 1789518932.677842 mapwrite 0x0001dc thru 0x00ce8c (0xccb1 bytes) 000318 1789518932.699977 punch from 0x021dc4 to 0x02380f (0x1a4c bytes) 000319 1789518932.705449 fallocate 0x031f4e thru 0x0385d7 (0x668a bytes) 000320 1789518932.713677 trunc from 0x040000 to 0x015b3f (0x2a4c0 bytes) 000321 1789518932.733409 trunc from 0x0138fc to 0x01456d (0xc72 bytes) 000322 1789518932.737958 mapwrite 0x02708e thru 0x0366a6 (0xf619 bytes) 000323 1789518932.808587 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x366a7): Mirror 4: 0xd59a0785 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366a7): Mirror 4: 0xd59a0785 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366a7: skipped 000324 1789518933.014995 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) 000325 1789518933.115930 trunc from 0x0366a7 to 0x027b6b (0xeb3b bytes) 000326 1789518933.140031 mapread 0x00a680 thru 0x00b341 (0xcc2 bytes) 000327 1789518933.156964 read 0x0019ed thru 0x002d27 (0x133b bytes) 000328 1789518933.163058 mapread 0x018b78 thru 0x024aa8 (0xbf31 bytes) 000329 1789518933.176912 fallocate 0x025bcf thru 0x030698 (0xaaca bytes) 000330 1789518933.180948 punch from 0x027025 to 0x027b6b (0xb47 bytes) 000331 1789518933.186271 punch from 0x01c351 to 0x0206d3 (0x4383 bytes) 000332 1789518933.193167 read 0x019b82 thru 0x020bb1 (0x7030 bytes) 000333 1789518933.212060 write 0x011aed thru 0x020f8f (0xf4a3 bytes) 000334 1789518933.215924 mapwrite 0x02c598 thru 0x0398e1 (0xd34a bytes) 000335 1789518933.271247 read 0x02db24 thru 0x0333a6 (0x5883 bytes) 000336 1789518933.271734 read 0x02a436 thru 0x02c343 (0x1f0e bytes) 000337 1789518933.272216 trunc from 0x0398e2 to 0x009cfa (0x2fbe7 bytes) 000338 1789518933.295475 mapwrite 0x02fef8 thru 0x03f2d3 (0xf3dc bytes) 000339 1789518933.338944 trunc from 0x0366e6 to 0x03f2d3 (0x8bee bytes) 000341 1789518933.347029 trunc from 0x0001bd to 0x00e3d1 (0xe215 bytes) 000342 1789518933.352672 mapread 0x000739 thru 0x0046c4 (0x3f8c bytes) 000344 1789518933.381571 trunc from 0x03f2d4 to 0x0021e9 (0x3d0ea bytes) 000346 1789518933.410929 write 0x03243b thru 0x03567e (0x3244 bytes) 000347 1789518933.412262 fallocate 0x01c8b3 thru 0x02597c (0x90ca bytes) 000348 1789518933.418555 read 0x0352c5 thru 0x03567e (0x3ba bytes) 000349 1789518933.419089 read 0x013868 thru 0x01d3a9 (0x9b42 bytes) 000350 1789518933.430298 trunc from 0x015327 to 0x01a82e (0x5508 bytes) 000351 1789518933.434307 trunc from 0x0044c3 to 0x00ba00 (0x753e bytes) 000352 1789518933.438715 trunc from 0x03567f to 0x02c091 (0x95ed bytes) 000354 1789518933.452826 trunc from 0x02c092 to 0x02c751 (0x6c0 bytes) 000356 1789518933.487159 trunc from 0x00650e to 0x008a34 (0x2527 bytes) 000357 1789518933.492496 trunc from 0x020d13 to 0x02c751 (0xba3f bytes) 000358 1789518933.496738 trunc from 0x0265ee to 0x02c751 (0x6164 bytes) 000359 1789518933.503801 trunc from 0x02c752 to 0x01add2 (0x1197f bytes) 000361 1789518933.523565 trunc from 0x01add3 to 0x02121e (0x644c bytes) 000362 1789518933.540522 read 0x012577 thru 0x01702d (0x4ab7 bytes) 000364 1789518933.553866 mapwrite 0x017f83 thru 0x019915 (0x1993 bytes) 000365 1789518933.569866 punch from 0x002754 to 0x007ee6 (0x5793 bytes) 000367 1789518933.574506 punch from 0x0208fa to 0x02121e (0x925 bytes) 000368 1789518933.578771 trunc from 0x0016bd to 0x005c51 (0x4595 bytes) 000369 1789518933.584491 punch from 0x00aeb9 to 0x017aff (0xcc47 bytes) 000370 1789518933.592022 MIRROR_EXTEND to 2 mirrors 000371 1789518934.089939 write 0x03174f thru 0x034207 (0x2ab9 bytes) 000372 1789518934.143581 trunc from 0x00c3f9 to 0x01278e (0x6396 bytes) 000373 1789518934.153459 fallocate 0x007d67 thru 0x00ef4b (0x71e5 bytes) 000374 1789518934.165052 punch from 0x01fc54 to 0x0298bf (0x9c6c bytes) 000375 1789518934.174142 read 0x02d219 thru 0x034207 (0x6fef bytes) 000378 1789518934.184225 trunc from 0x034208 to 0x013532 (0x20cd5 bytes) 000379 1789518934.233293 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x13533): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13533): Mirror 2: 0xc4454ef3 (stale) CRC-32 checksum value for chunk [0, 0x13533): Mirror 1: 0xc4454ef3 Mirror 3: 0xc4454ef3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13533: skipped 000380 1789518934.848340 fallocate 0x03060e thru 0x033061 (0x2a54 bytes) 000381 1789518934.879891 mapread 0x0258da thru 0x02fba7 (0xa2ce bytes) 000382 1789518934.896894 write 0x03bedf thru 0x03e4ab (0x25cd bytes) 000384 1789518934.901784 mapwrite 0x00d98d thru 0x012e1b (0x548f bytes) 000385 1789518934.929089 trunc from 0x03e4ac to 0x02b137 (0x13374 bytes) 000386 1789518934.982625 punch from 0x01d583 to 0x02b137 (0xdbb5 bytes) 000387 1789518934.990294 trunc from 0x020c1c to 0x02a104 (0x94e9 bytes) 000388 1789518934.997022 mapwrite 0x0348f8 thru 0x03ffff (0xb708 bytes) 000389 1789518935.046741 trunc from 0x015dc6 to 0x0211e7 (0xb422 bytes) 000390 1789518935.054723 trunc from 0x026088 to 0x031f40 (0xbeb9 bytes) 000392 1789518935.063993 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000393 1789518935.283528 mapread 0x018f42 thru 0x01b76e (0x282d bytes) 000394 1789518935.306626 mapread 0x00990f thru 0x0101b5 (0x68a7 bytes) 000395 1789518935.318239 mapwrite 0x024218 thru 0x025c21 (0x1a0a bytes) 000396 1789518935.339666 mapread 0x03e9b8 thru 0x03ffff (0x1648 bytes) 000397 1789518935.346845 write 0x01385c thru 0x019d24 (0x64c9 bytes) 000398 1789518935.360701 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xef41d338 Mirror 4: 0xef41d338 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000399 1789518936.048288 trunc from 0x040000 to 0x033494 (0xcb6b bytes) 000400 1789518936.114759 mapwrite 0x026620 thru 0x031711 (0xb0f2 bytes) 000401 1789518936.167561 write 0x03e9f9 thru 0x03ffff (0x1607 bytes) 000402 1789518936.168721 write 0x0350dc thru 0x03ffff (0xaf24 bytes) 000403 1789518936.181465 fallocate 0x01d4ad thru 0x020c8d (0x37e1 bytes) 000404 1789518936.199678 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xef41d338 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x43fef6f8 Mirror 5: 0x43fef6f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000405 1789518936.828862 mapwrite 0x00f5c9 thru 0x0134d5 (0x3f0d bytes) 000406 1789518936.909372 trunc from 0x00c034 to 0x015e00 (0x9dcd bytes) 000407 1789518936.913838 trunc from 0x040000 to 0x021f64 (0x1e09b bytes) 000408 1789518936.949088 read 0x01b1aa thru 0x01f09c (0x3ef3 bytes) 000409 1789518936.957760 mapwrite 0x02a0c3 thru 0x038c1f (0xeb5d bytes) 000410 1789518937.010722 punch from 0x013231 to 0x018192 (0x4f62 bytes) 000412 1789518937.017395 write 0x002475 thru 0x00a04b (0x7bd7 bytes) 000413 1789518937.025034 trunc from 0x013520 to 0x01e14e (0xac2f bytes) 000414 1789518937.029225 trunc from 0x002201 to 0x00a72e (0x852e bytes) 000415 1789518937.043469 trunc from 0x00e05e to 0x012be9 (0x4b8c bytes) 000416 1789518937.049058 write 0x012487 thru 0x020dcf (0xe949 bytes) 000417 1789518937.060332 write 0x03c811 thru 0x03ffff (0x37ef bytes) 000418 1789518937.061878 fallocate 0x0068d4 thru 0x011b9b (0xb2c8 bytes) 000419 1789518937.066076 fallocate 0x03a6dd thru 0x03e497 (0x3dbb bytes) 000420 1789518937.072264 trunc from 0x02c438 to 0x035686 (0x924f bytes) 000421 1789518937.078897 mapwrite 0x039f6b thru 0x03f909 (0x599f bytes) 000422 1789518937.113342 read 0x011f2d thru 0x01aa5c (0x8b30 bytes) 000423 1789518937.124153 trunc from 0x03f44a to 0x03ffff (0xbb6 bytes) 000424 1789518937.132578 mapread 0x02e352 thru 0x03555a (0x7209 bytes) 000425 1789518937.133619 read 0x0387cc thru 0x03ffff (0x7834 bytes) 000426 1789518937.139619 mapwrite 0x012c94 thru 0x01be2b (0x9198 bytes) 000427 1789518937.156748 write 0x024940 thru 0x027b46 (0x3207 bytes) 000428 1789518937.157996 mapwrite 0x0038c0 thru 0x00669d (0x2dde bytes) 000429 1789518937.180134 fallocate 0x00dae8 thru 0x00e333 (0x84c bytes) 000430 1789518937.185045 read 0x02603e thru 0x02856e (0x2531 bytes) 000431 1789518937.186518 punch from 0x02a0b3 to 0x0300d0 (0x601e bytes) 000432 1789518937.190900 mapwrite 0x002274 thru 0x00f3df (0xd16c bytes) 000433 1789518937.210264 mapwrite 0x01e53b thru 0x023fd1 (0x5a97 bytes) 000434 1789518937.230952 fallocate 0x037085 thru 0x03ffff (0x8f7b bytes) 000435 1789518937.240026 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 5: 0x43fef6f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 5: 0x43fef6f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000436 1789518937.488509 punch from 0x03c53b to 0x03ffff (0x3ac5 bytes) 000437 1789518937.494963 write 0x0204e7 thru 0x030130 (0xfc4a bytes) 000438 1789518937.513049 mapread 0x018016 thru 0x021f4b (0x9f36 bytes) 000439 1789518937.536048 punch from 0x0200e8 to 0x02d9ed (0xd906 bytes) 000440 1789518937.547208 punch from 0x01a690 to 0x027c52 (0xd5c3 bytes) 000441 1789518937.553951 mapread 0x03f5a9 thru 0x03ffff (0xa57 bytes) 000442 1789518937.563564 punch from 0x02ef22 to 0x03c907 (0xd9e6 bytes) 000443 1789518937.570313 punch from 0x015f74 to 0x017b2a (0x1bb7 bytes) 000444 1789518937.579703 MIRROR_RESYNC 3 mirrors 000445 1789518938.386170 mapread 0x0054bd thru 0x007a2c (0x2570 bytes) 000446 1789518938.403542 trunc from 0x015913 to 0x01f2b2 (0x99a0 bytes) 000447 1789518938.426944 write 0x037795 thru 0x03c92c (0x5198 bytes) 000448 1789518938.434311 mapread 0x01d9eb thru 0x021c93 (0x42a9 bytes) 000449 1789518938.440572 fallocate 0x02fef1 thru 0x033de6 (0x3ef6 bytes) 000450 1789518938.444504 write 0x01a939 thru 0x01ab19 (0x1e1 bytes) 000452 1789518938.445899 mapread 0x013961 thru 0x01a114 (0x67b4 bytes) 000453 1789518938.455158 write 0x024259 thru 0x031659 (0xd401 bytes) 000454 1789518938.461442 mapread 0x03be2c thru 0x03fc30 (0x3e05 bytes) 000455 1789518938.462389 fallocate 0x015c4a thru 0x019158 (0x350f bytes) 000456 1789518938.467047 read 0x002e60 thru 0x0109d0 (0xdb71 bytes) 000457 1789518938.474649 fallocate 0x006216 thru 0x00891d (0x2708 bytes) 000458 1789518938.480269 trunc from 0x040000 to 0x01c7e4 (0x2381b bytes) 000459 1789518938.500510 trunc from 0x00472a to 0x00e189 (0x9a60 bytes) 000460 1789518938.508027 read 0x010f90 thru 0x017b9f (0x6c10 bytes) 000461 1789518938.508775 trunc from 0x002f9a to 0x00537d (0x23e4 bytes) 000463 1789518938.513984 mapwrite 0x008db1 thru 0x010d92 (0x7fe2 bytes) 000464 1789518938.535732 punch from 0x01a559 to 0x01c7e4 (0x228c bytes) 000465 1789518938.540436 read 0x00150c thru 0x00e03c (0xcb31 bytes) 000466 1789518938.541444 trunc from 0x002a71 to 0x006cf5 (0x4285 bytes) 000467 1789518938.546175 mapwrite 0x02d88f thru 0x039bd7 (0xc349 bytes) 000468 1789518938.605451 write 0x026a97 thru 0x033e6b (0xd3d5 bytes) 000469 1789518938.613431 MIRROR_RESYNC 3 mirrors 000470 1789518939.235522 mapwrite 0x0277d6 thru 0x032fb4 (0xb7df bytes) 000471 1789518939.312463 mapread 0x023a32 thru 0x025998 (0x1f67 bytes) 000472 1789518939.313443 read 0x008ab7 thru 0x015673 (0xcbbd bytes) 000473 1789518939.322177 write 0x01465a thru 0x02356b (0xef12 bytes) 000474 1789518939.324024 trunc from 0x01b65a to 0x02a664 (0xf00b bytes) 000475 1789518939.332644 punch from 0x0299fc to 0x033383 (0x9988 bytes) 000476 1789518939.336781 mapread 0x0386ac thru 0x039bd7 (0x152c bytes) 000477 1789518939.337639 fallocate 0x00b29e thru 0x01212d (0x6e90 bytes) 000478 1789518939.341216 fallocate 0x014aa5 thru 0x0188b3 (0x3e0f bytes) 000479 1789518939.345561 trunc from 0x039bd8 to 0x0063d0 (0x33807 bytes) 000480 1789518939.358906 punch from 0x002b85 to 0x0063d0 (0x384c bytes) 000481 1789518939.363302 punch from 0x003a5a to 0x0063d0 (0x2977 bytes) 000482 1789518939.367595 trunc from 0x004f41 to 0x0063d0 (0x1490 bytes) 000483 1789518939.373009 trunc from 0x0063d1 to 0x00da9d (0x76cd bytes) 000484 1789518939.385047 mapwrite 0x026c6a thru 0x0340c7 (0xd45e bytes) 000485 1789518939.431802 trunc from 0x008e8f to 0x012169 (0x92db bytes) 000486 1789518939.436316 mapread 0x019543 thru 0x01e99b (0x5459 bytes) 000487 1789518939.442904 read 0x00622c thru 0x015e20 (0xfbf5 bytes) 000488 1789518939.449800 read 0x02450c thru 0x0332e1 (0xedd6 bytes) 000489 1789518939.453393 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x340c8): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340c8): Mirror 3: 0x2efaf0ff (stale) Mirror 5: 0x2efaf0ff (stale) CRC-32 checksum value for chunk [0, 0x340c8): Mirror 1: 0x17e7e4b5 Mirror 6: 0x17e7e4b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340c8: skipped 000490 1789518939.984738 MIRROR_RESYNC 4 mirrors 000491 1789518940.546985 fallocate 0x03f646 thru 0x03ffff (0x9ba bytes) 000492 1789518940.583428 MIRROR_RESYNC 4 mirrors 000493 1789518941.241260 fallocate 0x005733 thru 0x0098bb (0x4189 bytes) 000494 1789518941.277730 fallocate 0x02ff6a thru 0x035342 (0x53d9 bytes) 000495 1789518941.281531 fallocate 0x000b59 thru 0x00f89e (0xed46 bytes) 000496 1789518941.285610 trunc from 0x0340c8 to 0x02eb7a (0x554d bytes) 000497 1789518941.310579 trunc from 0x02eb7b to 0x02626c (0x890e bytes) 000498 1789518941.326168 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2626d): Mirror 5: 0x2d24b8a (stale) Mirror 6: 0x2d24b8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2626d): Mirror 5: 0x2d24b8a (stale) Mirror 6: 0x2d24b8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2626d: skipped 000500 1789518941.558307 read 0x016849 thru 0x019afc (0x32b4 bytes) 000502 1789518941.564386 trunc from 0x00da50 to 0x017c77 (0xa228 bytes) 000503 1789518941.572193 MIRROR_RESYNC 3 mirrors 000505 1789518942.044841 trunc from 0x02626d to 0x03feb9 (0x19c4d bytes) 000506 1789518942.102386 mapwrite 0x029243 thru 0x03769f (0xe45d bytes) 000507 1789518942.145220 write 0x03b414 thru 0x03ffff (0x4bec bytes) 000508 1789518942.146433 fallocate 0x006fcb thru 0x007519 (0x54f bytes) 000509 1789518942.154481 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc9545a44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc9545a44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000510 1789518942.404292 trunc from 0x040000 to 0x00a34b (0x35cb4 bytes) 000513 1789518942.422971 read 0x0005b4 thru 0x00a34b (0x9d98 bytes) 000515 1789518942.432142 mapwrite 0x015ebc thru 0x01a28d (0x43d2 bytes) 000516 1789518942.485407 mapwrite 0x018990 thru 0x018dbd (0x42e bytes) 000517 1789518942.504737 write 0x004d60 thru 0x008e54 (0x40f5 bytes) 000518 1789518942.505778 trunc from 0x01891c to 0x01a28d (0x1972 bytes) 000519 1789518942.511994 trunc from 0x01a28e to 0x0033a7 (0x16ee6 bytes) 000520 1789518942.545835 write 0x00f405 thru 0x014f29 (0x5b25 bytes) 000522 1789518942.548071 mapread 0x0042ae thru 0x00f3ea (0xb13d bytes) 000523 1789518942.555740 trunc from 0x00dd71 to 0x00f2b5 (0x1545 bytes) 000526 1789518942.571460 read 0x0066ce thru 0x00b634 (0x4f67 bytes) 000527 1789518942.572077 write 0x01678a thru 0x01c37e (0x5bf5 bytes) 000528 1789518942.573760 write 0x0302ec thru 0x03cbe9 (0xc8fe bytes) 000529 1789518942.576036 trunc from 0x03cbea to 0x0300f5 (0xcaf4 bytes) 000530 1789518942.589431 mapwrite 0x03234f thru 0x038b65 (0x6817 bytes) 000531 1789518942.634770 trunc from 0x038b66 to 0x00fb1d (0x29048 bytes) 000532 1789518942.655521 MIRROR_RESYNC 2 mirrors 000533 1789518943.154225 write 0x0311fd thru 0x038a26 (0x782a bytes) 000534 1789518943.187312 write 0x010d36 thru 0x015c6c (0x4f37 bytes) 000536 1789518943.196919 punch from 0x023fac to 0x02b47c (0x74d1 bytes) 000537 1789518943.202349 punch from 0x019fbb to 0x022aac (0x8af2 bytes) 000538 1789518943.208501 mapread 0x00d775 thru 0x014ca3 (0x752f bytes) 000539 1789518943.227420 punch from 0x035ca0 to 0x038a26 (0x2d87 bytes) 000540 1789518943.241495 trunc from 0x038a27 to 0x03d777 (0x4d51 bytes) 000541 1789518943.280990 mapwrite 0x03ec4b thru 0x03ffff (0x13b5 bytes) 000542 1789518943.316493 punch from 0x021567 to 0x02529b (0x3d35 bytes) 000543 1789518943.324127 fallocate 0x005950 thru 0x010c8a (0xb33b bytes) 000544 1789518943.328215 trunc from 0x040000 to 0x03a278 (0x5d87 bytes) 000545 1789518943.345728 trunc from 0x001471 to 0x001a40 (0x5d0 bytes) 000546 1789518943.350042 fallocate 0x020721 thru 0x0223d6 (0x1cb6 bytes) 000547 1789518943.358949 trunc from 0x01df0a to 0x029bae (0xbca5 bytes) 000548 1789518943.366877 read 0x01ced8 thru 0x01e123 (0x124c bytes) 000549 1789518943.373774 trunc from 0x0367fe to 0x03a278 (0x3a7b bytes) 000550 1789518943.378477 punch from 0x01f953 to 0x029853 (0x9f01 bytes) 000551 1789518943.383863 mapread 0x015609 thru 0x017533 (0x1f2b bytes) 000552 1789518943.386420 trunc from 0x03a279 to 0x0324e7 (0x7d91 bytes) 000553 1789518943.405333 write 0x030830 thru 0x037f25 (0x76f6 bytes) 000554 1789518943.412905 fallocate 0x015bb1 thru 0x02378d (0xdbdd bytes) 000555 1789518943.417543 read 0x03165a thru 0x037f25 (0x68cc bytes) 000556 1789518943.418086 read 0x02f1c7 thru 0x02f422 (0x25c bytes) 000557 1789518943.428583 trunc from 0x002533 to 0x0088dd (0x63ab bytes) 000558 1789518943.434691 punch from 0x02e720 to 0x0359df (0x72c0 bytes) 000559 1789518943.448436 fallocate 0x02c850 thru 0x038f5c (0xc70d bytes) 000561 1789518943.459886 MIRROR_RESYNC 2 mirrors 000562 1789518943.951818 mapread 0x02f0a9 thru 0x031cb2 (0x2c0a bytes) 000563 1789518943.968992 mapwrite 0x03def2 thru 0x03ffff (0x210e bytes) 000564 1789518944.062506 read 0x03dc0c thru 0x03ffff (0x23f4 bytes) 000566 1789518944.063956 write 0x01f5c2 thru 0x02032e (0xd6d bytes) 000567 1789518944.078036 punch from 0x012d81 to 0x021d29 (0xefa9 bytes) 000568 1789518944.091357 fallocate 0x0176f9 thru 0x024944 (0xd24c bytes) 000569 1789518944.095813 trunc from 0x040000 to 0x01d17f (0x22e80 bytes) 000570 1789518944.122540 trunc from 0x005ba6 to 0x0111b5 (0xb610 bytes) 000571 1789518944.127330 trunc from 0x017ca9 to 0x018e0a (0x1162 bytes) 000573 1789518944.132433 mapread 0x00822a thru 0x00c598 (0x436f bytes) 000574 1789518944.139089 mapwrite 0x00e298 thru 0x01296b (0x46d4 bytes) 000575 1789518944.161339 write 0x011732 thru 0x01f66b (0xdf3a bytes) 000576 1789518944.163215 punch from 0x01ba76 to 0x01f66b (0x3bf6 bytes) 000577 1789518944.181530 fallocate 0x01c587 thru 0x022a5f (0x64d9 bytes) 000579 1789518944.187584 read 0x010e66 thru 0x019b41 (0x8cdc bytes) 000580 1789518944.188075 write 0x02b4d9 thru 0x030d16 (0x583e bytes) 000581 1789518944.189402 trunc from 0x00f19c to 0x01d289 (0xe0ee bytes) 000582 1789518944.196291 mapwrite 0x0047b7 thru 0x007700 (0x2f4a bytes) 000583 1789518944.216014 write 0x0392a7 thru 0x039591 (0x2eb bytes) 000584 1789518944.217425 punch from 0x003109 to 0x00fcc4 (0xcbbc bytes) 000585 1789518944.232894 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) 000586 1789518944.341528 punch from 0x01fa1b to 0x0251c3 (0x57a9 bytes) 000587 1789518944.349032 mapread 0x013e72 thru 0x02207d (0xe20c bytes) 000588 1789518944.385354 fallocate 0x0385b2 thru 0x03ffff (0x7a4e bytes) 000589 1789518944.394936 punch from 0x014323 to 0x017c22 (0x3900 bytes) 000590 1789518944.401870 mapread 0x039c9e thru 0x03b959 (0x1cbc bytes) 000591 1789518944.411577 mapread 0x03e87a thru 0x03ffff (0x1786 bytes) 000592 1789518944.412537 trunc from 0x040000 to 0x006dc5 (0x3923a bytes) 000593 1789518944.431013 trunc from 0x003613 to 0x006dc5 (0x37b3 bytes) 000595 1789518944.439376 trunc from 0x006dc6 to 0x03d866 (0x36aa1 bytes) 000596 1789518944.458152 mapread 0x00ca8d thru 0x00fa2f (0x2fa3 bytes) 000597 1789518944.465761 punch from 0x024266 to 0x02e129 (0x9ec4 bytes) 000598 1789518944.470791 mapread 0x029ffa thru 0x02dc0c (0x3c13 bytes) 000599 1789518944.482741 trunc from 0x03296a to 0x032c82 (0x319 bytes) 000600 1789518944.488762 read 0x00c9b3 thru 0x00fd4c (0x339a bytes) 000601 1789518944.489442 trunc from 0x03d867 to 0x02847b (0x153eb bytes) 000602 1789518944.507393 mapwrite 0x019057 thru 0x01f676 (0x6620 bytes) 000604 1789518944.534510 trunc from 0x02847c to 0x032416 (0x9f9b bytes) 000605 1789518944.549267 fallocate 0x0397fb thru 0x03ce5c (0x3662 bytes) 000607 1789518944.557420 trunc from 0x02c013 to 0x032416 (0x6404 bytes) 000608 1789518944.563045 mapwrite 0x025997 thru 0x02883f (0x2ea9 bytes) 000609 1789518944.575636 read 0x020da3 thru 0x02bc59 (0xaeb7 bytes) 000610 1789518944.583090 punch from 0x007b3a to 0x012299 (0xa760 bytes) 000611 1789518944.587179 mapread 0x0121e3 thru 0x0174bc (0x52da bytes) 000613 1789518944.595063 punch from 0x00132b to 0x010960 (0xf636 bytes) 000615 1789518944.604875 mapwrite 0x003b34 thru 0x01240b (0xe8d8 bytes) 000617 1789518944.708030 trunc from 0x032417 to 0x0209db (0x11a3b bytes) 000618 1789518944.726709 mapread 0x017c9d thru 0x0209db (0x8d3f bytes) 000619 1789518944.727716 mapread 0x0003a3 thru 0x00f767 (0xf3c5 bytes) 000620 1789518944.738216 fallocate 0x0143fb thru 0x020052 (0xbc58 bytes) 000621 1789518944.742801 mapread 0x00940c thru 0x019346 (0xff3b bytes) 000622 1789518944.744092 trunc from 0x0209dc to 0x0129aa (0xe031 bytes) 000623 1789518944.760898 trunc from 0x0129ab to 0x034586 (0x21bdc bytes) 000624 1789518944.780955 mapwrite 0x0218dc thru 0x02f208 (0xd92d bytes) 000625 1789518944.877839 punch from 0x01e7fc to 0x025c4e (0x7453 bytes) 000626 1789518944.883152 mapread 0x002579 thru 0x00d2cb (0xad53 bytes) 000627 1789518944.883995 trunc from 0x034587 to 0x01343f (0x21147 bytes) 000630 1789518944.907758 punch from 0x01258f to 0x01343f (0xeb1 bytes) 000631 1789518944.913806 write 0x03dac6 thru 0x03ffff (0x253a bytes) 000632 1789518944.915124 trunc from 0x040000 to 0x0289d4 (0x1762b bytes) 000633 1789518944.925301 write 0x015e25 thru 0x021e7c (0xc058 bytes) 000634 1789518944.935960 MIRROR_EXTEND to 2 mirrors 000635 1789518945.532257 write 0x016b96 thru 0x01c64a (0x5ab5 bytes) 000636 1789518945.592805 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x289d5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x289d5): Mirror 2: 0xa60e6b0e (stale) CRC-32 checksum value for chunk [0, 0x289d5): Mirror 1: 0x3ff891b1 Mirror 3: 0x3ff891b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x289d5: skipped 000637 1789518946.330989 mapread 0x02572e thru 0x0289d4 (0x32a7 bytes) 000638 1789518946.360488 fallocate 0x03ccda thru 0x03ffff (0x3326 bytes) 000639 1789518946.391844 mapread 0x02ee3a thru 0x03d1c4 (0xe38b bytes) 000640 1789518946.419050 write 0x014e04 thru 0x01569c (0x899 bytes) 000641 1789518946.457077 MIRROR_RESYNC 3 mirrors 000643 1789518947.237424 fallocate 0x002851 thru 0x0064a1 (0x3c51 bytes) 000644 1789518947.266179 mapwrite 0x00e1ca thru 0x016297 (0x80ce bytes) 000645 1789518947.309470 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x71c41fd8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x71c41fd8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000646 1789518947.563058 read 0x01ea8b thru 0x021a6e (0x2fe4 bytes) 000647 1789518947.569481 mapwrite 0x0061a3 thru 0x00d364 (0x71c2 bytes) 000649 1789518947.598327 punch from 0x028d4f to 0x02a629 (0x18db bytes) 000650 1789518947.602481 mapread 0x017cfa thru 0x01a61c (0x2923 bytes) 000651 1789518947.611047 trunc from 0x040000 to 0x019a11 (0x265ee bytes) 000653 1789518947.630937 trunc from 0x00c9de to 0x00f1a1 (0x27c4 bytes) 000654 1789518947.634875 punch from 0x0194b5 to 0x019a11 (0x55d bytes) 000655 1789518947.641529 trunc from 0x019a12 to 0x007c8c (0x11d85 bytes) 000656 1789518947.656605 write 0x01a1ec thru 0x01c088 (0x1e9d bytes) 000658 1789518947.666041 MIRROR_RESYNC 2 mirrors 000659 1789518948.167504 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) 000661 1789518948.288743 punch from 0x01a1c0 to 0x01c088 (0x1ec9 bytes) 000662 1789518948.301209 mapwrite 0x005b60 thru 0x008738 (0x2bd9 bytes) 000664 1789518948.327029 trunc from 0x001fe9 to 0x011d0c (0xfd24 bytes) 000665 1789518948.332681 trunc from 0x01c089 to 0x01893b (0x374d bytes) 000666 1789518948.375937 punch from 0x012076 to 0x013f10 (0x1e9b bytes) 000667 1789518948.382074 mapwrite 0x017133 thru 0x0226c3 (0xb591 bytes) 000668 1789518948.429466 trunc from 0x0226c4 to 0x02855b (0x5e98 bytes) 000669 1789518948.455237 trunc from 0x024d8b to 0x02855b (0x37d1 bytes) 000670 1789518948.462831 mapread 0x012266 thru 0x01f03d (0xcdd8 bytes) 000671 1789518948.466490 trunc from 0x008fe3 to 0x00da3d (0x4a5b bytes) 000672 1789518948.471683 fallocate 0x00f0fd thru 0x014dd1 (0x5cd5 bytes) 000673 1789518948.479159 write 0x016a00 thru 0x017067 (0x668 bytes) 000674 1789518948.480056 punch from 0x028381 to 0x02855b (0x1db bytes) 000675 1789518948.484806 fallocate 0x02016a thru 0x02935e (0x91f5 bytes) 000677 1789518948.493627 write 0x0034da thru 0x0093d0 (0x5ef7 bytes) 000678 1789518948.508937 punch from 0x01a4b5 to 0x01fd35 (0x5881 bytes) 000680 1789518948.517784 trunc from 0x00524a to 0x00b533 (0x62ea bytes) 000681 1789518948.530713 mapwrite 0x01afc5 thru 0x0288b0 (0xd8ec bytes) 000682 1789518948.566406 read 0x0064c3 thru 0x007b94 (0x16d2 bytes) 000683 1789518948.572814 write 0x00052f thru 0x00c3a3 (0xbe75 bytes) 000684 1789518948.586811 fallocate 0x000435 thru 0x0099c7 (0x9593 bytes) 000686 1789518948.592682 read 0x00dfae thru 0x013f13 (0x5f66 bytes) 000687 1789518948.599971 trunc from 0x00675c to 0x00d6df (0x6f84 bytes) 000688 1789518948.613254 trunc from 0x02935f to 0x019d86 (0xf5d8 bytes) 000689 1789518948.627867 punch from 0x00aaa1 to 0x01391e (0x8e7e bytes) 000691 1789518948.635530 trunc from 0x019d87 to 0x010eff (0x8e87 bytes) 000692 1789518948.657803 trunc from 0x010f00 to 0x01b810 (0xa911 bytes) 000693 1789518948.675371 mapread 0x008093 thru 0x00f4bd (0x742b bytes) 000694 1789518948.716270 trunc from 0x01b811 to 0x03d28f (0x21a7f bytes) 000695 1789518948.729633 read 0x014429 thru 0x018399 (0x3f71 bytes) 000697 1789518948.742801 punch from 0x011343 to 0x01ce94 (0xbb52 bytes) 000698 1789518948.747755 trunc from 0x00f0b9 to 0x0150bc (0x6004 bytes) 000699 1789518948.752142 fallocate 0x01a640 thru 0x020082 (0x5a43 bytes) 000702 1789518948.758125 read 0x02ef89 thru 0x034b5c (0x5bd4 bytes) 000703 1789518948.758836 trunc from 0x02b945 to 0x031ba7 (0x6263 bytes) 000704 1789518948.768839 trunc from 0x021010 to 0x02c0d2 (0xb0c3 bytes) 000706 1789518948.780448 mapwrite 0x01d054 thru 0x02c391 (0xf33e bytes) 000707 1789518948.822121 fallocate 0x02ca3b thru 0x035897 (0x8e5d bytes) 000708 1789518948.826432 write 0x031c18 thru 0x03567e (0x3a67 bytes) 000709 1789518948.827804 trunc from 0x01fafc to 0x01fc4b (0x150 bytes) 000710 1789518948.832993 mapwrite 0x000824 thru 0x005f61 (0x573e bytes) 000711 1789518948.862573 write 0x002117 thru 0x00ce71 (0xad5b bytes) 000712 1789518948.864073 trunc from 0x0307cb to 0x03d28f (0xcac5 bytes) 000713 1789518948.869966 trunc from 0x03d290 to 0x013a34 (0x2985b bytes) 000714 1789518948.886504 read 0x00a7ee thru 0x013a34 (0x9247 bytes) 000716 1789518948.888270 write 0x0190aa thru 0x0271da (0xe131 bytes) 000717 1789518948.891901 read 0x009fa5 thru 0x01550f (0xb56b bytes) 000718 1789518948.898599 mapread 0x024714 thru 0x0271da (0x2ac7 bytes) 000719 1789518948.900198 fallocate 0x0103c3 thru 0x012d4d (0x298b bytes) 000720 1789518948.906023 mapread 0x015411 thru 0x021a4e (0xc63e bytes) 000721 1789518948.907141 punch from 0x0144a6 to 0x01bcf9 (0x7854 bytes) 000722 1789518948.918316 trunc from 0x0271db to 0x013844 (0x13996 bytes) 000723 1789518948.935024 trunc from 0x013845 to 0x02b72b (0x17ee7 bytes) 000725 1789518948.953649 fallocate 0x0168f7 thru 0x0199d2 (0x30dc bytes) 000726 1789518948.960906 trunc from 0x016b9e to 0x01b9a6 (0x4e09 bytes) 000727 1789518948.964946 write 0x02f120 thru 0x03006c (0xf4d bytes) 000728 1789518948.966871 read 0x0159d1 thru 0x01f2eb (0x991b bytes) 000729 1789518948.975465 mapread 0x00d0b9 thru 0x018b9d (0xbae5 bytes) 000730 1789518948.976554 punch from 0x01e85a to 0x02bd41 (0xd4e8 bytes) 000731 1789518948.983311 write 0x00e876 thru 0x01b655 (0xcde0 bytes) 000732 1789518948.988460 write 0x0374da thru 0x03ffff (0x8b26 bytes) 000736 1789518948.997631 mapwrite 0x032cf2 thru 0x0392c9 (0x65d8 bytes) 000737 1789518949.031125 trunc from 0x040000 to 0x00fe88 (0x30177 bytes) 000738 1789518949.049119 trunc from 0x00fe89 to 0x033ff5 (0x2416d bytes) 000739 1789518949.062962 mapwrite 0x01b3e5 thru 0x021f0a (0x6b26 bytes) 000740 1789518949.123374 trunc from 0x0236b1 to 0x02c2e3 (0x8c33 bytes) 000741 1789518949.129163 mapwrite 0x021439 thru 0x02c40c (0xafd4 bytes) 000743 1789518949.217482 mapread 0x0040a6 thru 0x011d58 (0xdcb3 bytes) 000744 1789518949.231223 trunc from 0x033ff6 to 0x02f90e (0x46e7 bytes) 000745 1789518949.249046 write 0x017e80 thru 0x01a152 (0x22d3 bytes) 000746 1789518949.259401 punch from 0x01718e to 0x020e5c (0x9ccf bytes) 000747 1789518949.272980 trunc from 0x02f90f to 0x03a511 (0xac03 bytes) 000748 1789518949.288384 write 0x01cdab thru 0x02388e (0x6ae4 bytes) 000749 1789518949.296712 punch from 0x01ccc9 to 0x025df4 (0x912c bytes) 000750 1789518949.311911 fallocate 0x036ac8 thru 0x03e903 (0x7e3c bytes) 000751 1789518949.316174 trunc from 0x03e904 to 0x01ba20 (0x22ee3 bytes) 000752 1789518949.336168 read 0x00483f thru 0x00d0c3 (0x8885 bytes) 000753 1789518949.339230 fallocate 0x01f64b thru 0x02dde8 (0xe79e bytes) 000755 1789518949.348603 read 0x00f1cd thru 0x0182b2 (0x90e6 bytes) 000757 1789518949.367676 read 0x00651b thru 0x00d4c4 (0x6faa bytes) 000758 1789518949.368753 trunc from 0x01ba21 to 0x01c5fd (0xbdd bytes) 000759 1789518949.387929 punch from 0x004ede to 0x00d02e (0x8151 bytes) 000763 1789518949.393762 write 0x032211 thru 0x03df43 (0xbd33 bytes) 000764 1789518949.396595 trunc from 0x03df44 to 0x0072bd (0x36c86 bytes) 000765 1789518949.413296 write 0x02fba1 thru 0x0319d3 (0x1e33 bytes) 000766 1789518949.414600 mapwrite 0x0336bf thru 0x03db2d (0xa46f bytes) 000768 1789518949.456608 mapwrite 0x00c2b1 thru 0x0131cc (0x6f1c bytes) 000769 1789518949.481615 mapread 0x013d39 thru 0x01992e (0x5bf6 bytes) 000771 1789518949.487718 punch from 0x01d660 to 0x01e9bd (0x135e bytes) 000772 1789518949.495295 punch from 0x03721a to 0x037ad8 (0x8bf bytes) 000773 1789518949.499083 fallocate 0x022169 thru 0x028421 (0x62b9 bytes) 000774 1789518949.504031 mapwrite 0x03f814 thru 0x03ffff (0x7ec bytes) 000775 1789518949.539966 read 0x01bab9 thru 0x022755 (0x6c9d bytes) 000777 1789518949.549190 fallocate 0x029942 thru 0x02bd54 (0x2413 bytes) 000778 1789518949.555484 punch from 0x00d678 to 0x017bff (0xa588 bytes) 000780 1789518949.564245 MIRROR_EXTEND to 2 mirrors 000781 1789518950.159011 trunc from 0x040000 to 0x036f02 (0x90fd bytes) 000783 1789518950.212801 fallocate 0x00fbbb thru 0x01f4bb (0xf901 bytes) 000784 1789518950.217928 read 0x01afdd thru 0x0241f8 (0x921c bytes) 000785 1789518950.238796 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) 000786 1789518950.386182 fallocate 0x01033e thru 0x01542e (0x50f1 bytes) 000787 1789518950.398012 MIRROR_EXTEND to 2 mirrors 000789 1789518950.947854 write 0x0119bf thru 0x01fec1 (0xe503 bytes) 000790 1789518950.982743 write 0x015ea5 thru 0x018c61 (0x2dbd bytes) 000791 1789518950.987511 fallocate 0x02efd3 thru 0x03ab91 (0xbbbf bytes) 000792 1789518950.995432 write 0x0003cf thru 0x0090b1 (0x8ce3 bytes) 000793 1789518951.010967 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36f03): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f03): Mirror 2: 0x6d2ad0ec (stale) CRC-32 checksum value for chunk [0, 0x36f03): Mirror 1: 0x4e79a5b8 Mirror 3: 0x4e79a5b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f03: skipped 000794 1789518951.713309 trunc from 0x0089cb to 0x018892 (0xfec8 bytes) 000796 1789518951.757538 mapwrite 0x0134c3 thru 0x01f3f0 (0xbf2e bytes) 000797 1789518951.809247 mapread 0x00a21d thru 0x01828c (0xe070 bytes) 000798 1789518951.822429 mapread 0x02ac14 thru 0x036f02 (0xc2ef bytes) 000800 1789518951.838049 write 0x00b4f3 thru 0x00b875 (0x383 bytes) 000801 1789518951.842020 trunc from 0x023dcd to 0x028aa4 (0x4cd8 bytes) 000802 1789518951.846023 write 0x0107c0 thru 0x01d135 (0xc976 bytes) 000803 1789518951.847578 fallocate 0x038a24 thru 0x039443 (0xa20 bytes) 000804 1789518951.859834 mapwrite 0x02877c thru 0x0307f2 (0x8077 bytes) 000805 1789518951.909893 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36f03): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f03): Mirror 2: 0x6d2ad0ec (stale) Mirror 3: 0x4e79a5b8 (stale) CRC-32 checksum value for chunk [0, 0x36f03): Mirror 1: 0xbf4d38d4 Mirror 4: 0xbf4d38d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f03: skipped 000806 1789518952.677316 mapread 0x0300bb thru 0x0305b1 (0x4f7 bytes) 000807 1789518952.683937 mapwrite 0x00cc47 thru 0x016e43 (0xa1fd bytes) 000808 1789518952.773062 punch from 0x02953c to 0x036f02 (0xd9c7 bytes) 000809 1789518952.785136 mapread 0x02b372 thru 0x035463 (0xa0f2 bytes) 000811 1789518952.809653 fallocate 0x02ac7a thru 0x0311a4 (0x652b bytes) 000812 1789518952.815848 trunc from 0x036f03 to 0x03c58a (0x5688 bytes) 000813 1789518952.840700 fallocate 0x03e99c thru 0x03ffff (0x1664 bytes) 000814 1789518952.847011 mapwrite 0x01f983 thru 0x024cca (0x5348 bytes) 000815 1789518952.877732 read 0x01e545 thru 0x02a336 (0xbdf2 bytes) 000816 1789518952.878587 mapread 0x001334 thru 0x0076de (0x63ab bytes) 000817 1789518952.890797 trunc from 0x03c58b to 0x023b76 (0x18a14 bytes) 000818 1789518952.907879 mapread 0x010cbd thru 0x018340 (0x7684 bytes) 000819 1789518952.908974 read 0x00d8a8 thru 0x018e52 (0xb5ab bytes) 000820 1789518952.909637 trunc from 0x0086c8 to 0x01037e (0x7cb7 bytes) 000822 1789518952.923770 trunc from 0x023b77 to 0x018789 (0xb3ed bytes) 000823 1789518952.941819 fallocate 0x02d710 thru 0x035045 (0x7936 bytes) 000824 1789518952.949315 mapwrite 0x038cb7 thru 0x03c951 (0x3c9b bytes) 000825 1789518952.998149 mapread 0x0255a9 thru 0x02d709 (0x8161 bytes) 000826 1789518952.999228 trunc from 0x035e86 to 0x03c951 (0x6acc bytes) 000827 1789518953.010983 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c952): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c952): Mirror 2: 0x880c5474 (stale) Mirror 3: 0xd7f69ee1 (stale) Mirror 4: 0x22b882e7 (stale) CRC-32 checksum value for chunk [0, 0x3c952): Mirror 1: 0xe448af77 Mirror 5: 0xe448af77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c952: skipped 000829 1789518953.739928 mapwrite 0x01ba5b thru 0x01dd82 (0x2328 bytes) 000830 1789518953.825443 fallocate 0x03af99 thru 0x03ffff (0x5067 bytes) 000831 1789518953.832199 read 0x0323e7 thru 0x03ffff (0xdc19 bytes) 000832 1789518953.841460 mapwrite 0x033b91 thru 0x03d016 (0x9486 bytes) 000833 1789518953.868973 fallocate 0x0321f2 thru 0x03d487 (0xb296 bytes) 000834 1789518953.879016 write 0x01d8df thru 0x02cbb4 (0xf2d6 bytes) 000835 1789518953.890610 mapwrite 0x01213f thru 0x017b47 (0x5a09 bytes) 000836 1789518953.927957 mapwrite 0x032050 thru 0x03cc4e (0xabff bytes) 000837 1789518953.951569 punch from 0x03ace0 to 0x03ffff (0x5320 bytes) 000838 1789518953.957782 mapread 0x00458f thru 0x00b521 (0x6f93 bytes) 000839 1789518953.980212 punch from 0x01ec47 to 0x024853 (0x5c0d bytes) 000840 1789518953.998512 write 0x01c2b3 thru 0x01d307 (0x1055 bytes) 000841 1789518953.999476 punch from 0x003bf2 to 0x0046d7 (0xae6 bytes) 000842 1789518954.003770 punch from 0x03f599 to 0x03ffff (0xa67 bytes) 000843 1789518954.014020 trunc from 0x040000 to 0x008f27 (0x370d8 bytes) 000844 1789518954.044813 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x8f28): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8f28): Mirror 2: 0x1d2057c1 (stale) Mirror 3: 0xe448bf57 (stale) Mirror 4: 0xd53a6cbd (stale) Mirror 5: 0x5a759af0 (stale) CRC-32 checksum value for chunk [0, 0x8f28): Mirror 1: 0x89022a2e Mirror 6: 0x89022a2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8f28: skipped 000845 1789518954.688256 punch from 0x00261e to 0x007db0 (0x5793 bytes) 000846 1789518954.722533 trunc from 0x006f6f to 0x008f27 (0x1fb9 bytes) 000847 1789518954.730386 punch from 0x0058ac to 0x008f27 (0x367c bytes) 000849 1789518954.735880 fallocate 0x029ad4 thru 0x0374d8 (0xda05 bytes) 000850 1789518954.739550 mapread 0x002b50 thru 0x004966 (0x1e17 bytes) 000851 1789518954.754163 trunc from 0x008f28 to 0x01abfa (0x11cd3 bytes) 000852 1789518954.773907 read 0x00d0ee thru 0x014cf9 (0x7c0c bytes) 000853 1789518954.781481 write 0x005f66 thru 0x006bad (0xc48 bytes) 000854 1789518954.782429 write 0x0086b0 thru 0x00c87d (0x41ce bytes) 000855 1789518954.788496 write 0x016020 thru 0x01cdeb (0x6dcc bytes) 000856 1789518954.794539 write 0x0274e5 thru 0x03292d (0xb449 bytes) 000857 1789518954.796146 punch from 0x0302f1 to 0x031714 (0x1424 bytes) 000858 1789518954.808942 write 0x029bff thru 0x02e950 (0x4d52 bytes) 000859 1789518954.810473 mapwrite 0x021a52 thru 0x028b0b (0x70ba bytes) 000860 1789518954.838584 mapread 0x00a2b2 thru 0x00a54c (0x29b bytes) 000861 1789518954.839524 trunc from 0x0182ef to 0x018c2f (0x941 bytes) 000862 1789518954.846858 write 0x0055ae thru 0x007703 (0x2156 bytes) 000863 1789518954.847847 trunc from 0x02342d to 0x02a897 (0x746b bytes) 000864 1789518954.857656 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3292e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3292e): Mirror 2: 0x40bd7181 (stale) Mirror 3: 0xb72b6972 (stale) Mirror 4: 0x47b36630 (stale) Mirror 5: 0xde33ac7d (stale) Mirror 6: 0xca754584 (stale) CRC-32 checksum value for chunk [0, 0x3292e): Mirror 1: 0x20dbfd8b Mirror 7: 0x20dbfd8b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3292e: skipped 000865 1789518955.772862 mapread 0x028437 thru 0x030a89 (0x8653 bytes) 000866 1789518955.807272 punch from 0x011583 to 0x019835 (0x82b3 bytes) 000867 1789518955.838571 mapread 0x01e172 thru 0x02a89d (0xc72c bytes) 000869 1789518955.854874 mapread 0x004ff7 thru 0x00faf9 (0xab03 bytes) 000870 1789518955.867825 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3292e): Mirror 3: 0xb72b6972 (stale) Mirror 4: 0x47b36630 (stale) Mirror 5: 0xde33ac7d (stale) Mirror 6: 0xca754584 (stale) Mirror 7: 0x20dbfd8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3292e): Mirror 3: 0xb72b6972 (stale) Mirror 4: 0x47b36630 (stale) Mirror 5: 0xde33ac7d (stale) Mirror 6: 0xca754584 (stale) Mirror 7: 0x20dbfd8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3292e: skipped 000871 1789518956.188424 fallocate 0x034cbc thru 0x03d34f (0x8694 bytes) 000872 1789518956.192760 trunc from 0x03292e to 0x031d5d (0xbd0 bytes) 000873 1789518956.212875 mapread 0x0257ff thru 0x02e355 (0x8b57 bytes) 000874 1789518956.233205 trunc from 0x031d5e to 0x0367cd (0x4a70 bytes) 000875 1789518956.247458 punch from 0x0311fa to 0x0367cd (0x55d4 bytes) 000876 1789518956.252676 trunc from 0x0367ce to 0x029b96 (0xcc37 bytes) 000878 1789518956.270241 read 0x012d1d thru 0x01ab8b (0x7e6f bytes) 000879 1789518956.285280 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29b97): Mirror 4: 0xf4fb16cc (stale) Mirror 5: 0x984cb68c (stale) Mirror 6: 0x39e7f18f (stale) Mirror 7: 0x3f025b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b97): Mirror 4: 0xf4fb16cc (stale) Mirror 5: 0x984cb68c (stale) Mirror 6: 0x39e7f18f (stale) Mirror 7: 0x3f025b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b97: skipped 000880 1789518956.584267 punch from 0x014a33 to 0x023ee3 (0xf4b1 bytes) 000881 1789518956.589715 trunc from 0x029b97 to 0x02a491 (0x8fb bytes) 000882 1789518956.611252 trunc from 0x0115a3 to 0x01f61a (0xe078 bytes) 000883 1789518956.617033 write 0x02656c thru 0x03397c (0xd411 bytes) 000884 1789518956.622482 mapread 0x00d111 thru 0x019e21 (0xcd11 bytes) 000885 1789518956.652879 punch from 0x009517 to 0x00f5a4 (0x608e bytes) 000887 1789518956.659945 mapwrite 0x0268ed thru 0x02c3b0 (0x5ac4 bytes) 000888 1789518956.681520 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3397d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3397d): Mirror 4: 0xce2e89e1 (stale) Mirror 5: 0x2987dc7f (stale) Mirror 6: 0x46d924e5 (stale) Mirror 7: 0xaef5d413 (stale) CRC-32 checksum value for chunk [0, 0x3397d): Mirror 1: 0x3a283843 Mirror 8: 0x3a283843 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3397d: skipped 000889 1789518957.464365 MIRROR_RESYNC 6 mirrors 000890 1789518958.519811 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) 000891 1789518958.799266 trunc from 0x00074e to 0x00a8de (0xa191 bytes) 000892 1789518958.830647 MIRROR_RESYNC 5 mirrors 000893 1789518959.722066 write 0x03f7d5 thru 0x03ffff (0x82b bytes) 000894 1789518959.745010 write 0x001e0e thru 0x011422 (0xf615 bytes) 000895 1789518959.754488 mapwrite 0x0011e4 thru 0x00fdaf (0xebcc bytes) 000896 1789518959.780220 fallocate 0x005075 thru 0x00c270 (0x71fc bytes) 000897 1789518959.789032 read 0x03b721 thru 0x03ffff (0x48df bytes) 000898 1789518959.795864 trunc from 0x040000 to 0x034678 (0xb987 bytes) 000899 1789518959.818322 read 0x02b66a thru 0x034678 (0x900f bytes) 000901 1789518959.831366 trunc from 0x034679 to 0x024ec1 (0xf7b7 bytes) 000902 1789518959.858834 read 0x00fc06 thru 0x01b54e (0xb949 bytes) 000903 1789518959.859565 fallocate 0x018c25 thru 0x024d7f (0xc15b bytes) 000904 1789518959.864867 read 0x0137ea thru 0x01e125 (0xa93c bytes) 000905 1789518959.866215 trunc from 0x024ec2 to 0x024598 (0x929 bytes) 000906 1789518959.894831 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24599): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24599): Mirror 4: 0x40776a6f (stale) Mirror 5: 0x40776a6f (stale) Mirror 6: 0x40776a6f (stale) Mirror 7: 0x40776a6f (stale) CRC-32 checksum value for chunk [0, 0x24599): Mirror 1: 0xe60f59fe Mirror 8: 0xe60f59fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24599: skipped 000907 1789518960.638073 mapread 0x01b13b thru 0x01d3fd (0x22c3 bytes) 000908 1789518960.674356 trunc from 0x021e97 to 0x024598 (0x2702 bytes) 000909 1789518960.707583 read 0x01b84d thru 0x01d4b7 (0x1c6b bytes) 000910 1789518960.713677 write 0x010102 thru 0x01b663 (0xb562 bytes) 000911 1789518960.724335 mapwrite 0x0007f3 thru 0x0099bd (0x91cb bytes) 000913 1789518960.770152 mapread 0x018639 thru 0x02215f (0x9b27 bytes) 000914 1789518960.784855 trunc from 0x024599 to 0x005c2c (0x1e96c bytes) 000915 1789518960.806674 trunc from 0x005c2d to 0x012e8f (0xd263 bytes) 000916 1789518960.821560 mapwrite 0x018b1b thru 0x025f1a (0xd400 bytes) 000917 1789518960.883280 read 0x00e54d thru 0x010d07 (0x27bb bytes) 000919 1789518960.894430 trunc from 0x025f1b to 0x01eaa2 (0x7478 bytes) 000921 1789518960.907974 mapwrite 0x0380b9 thru 0x03ffff (0x7f47 bytes) 000922 1789518960.977882 read 0x01bb72 thru 0x020601 (0x4a90 bytes) 000924 1789518960.992985 mapread 0x0283bb thru 0x032a35 (0xa67b bytes) 000925 1789518960.994577 trunc from 0x040000 to 0x00cec0 (0x3313f bytes) 000926 1789518961.027665 write 0x037543 thru 0x03ffff (0x8abd bytes) 000927 1789518961.029174 punch from 0x00073f to 0x006af0 (0x63b2 bytes) 000928 1789518961.035029 mapread 0x00b13d thru 0x00b5f9 (0x4bd bytes) 000929 1789518961.035946 mapread 0x020034 thru 0x022eb6 (0x2e83 bytes) 000930 1789518961.054452 fallocate 0x006b09 thru 0x00c80d (0x5d05 bytes) 000931 1789518961.061735 fallocate 0x03fb92 thru 0x03ffff (0x46e bytes) 000932 1789518961.067871 trunc from 0x014c0b to 0x0247c3 (0xfbb9 bytes) 000933 1789518961.075001 punch from 0x02b7b2 to 0x03671e (0xaf6d bytes) 000934 1789518961.087322 mapwrite 0x009548 thru 0x0182de (0xed97 bytes) 000935 1789518961.176937 mapread 0x01a92d thru 0x027253 (0xc927 bytes) 000936 1789518961.225359 read 0x026171 thru 0x033d86 (0xdc16 bytes) 000938 1789518961.233888 punch from 0x02fc1c to 0x033f15 (0x42fa bytes) 000939 1789518961.237822 mapread 0x037e5d thru 0x03f377 (0x751b bytes) 000940 1789518961.238808 punch from 0x016966 to 0x024fa3 (0xe63e bytes) 000941 1789518961.243050 punch from 0x01add7 to 0x01d3ae (0x25d8 bytes) 000942 1789518961.248646 mapwrite 0x01cff1 thru 0x022d44 (0x5d54 bytes) 000943 1789518961.301528 write 0x03090e thru 0x0353f3 (0x4ae6 bytes) 000944 1789518961.310750 MIRROR_RESYNC 6 mirrors 000945 1789518962.819375 write 0x037b0e thru 0x03ffff (0x84f2 bytes) 000946 1789518962.851086 trunc from 0x040000 to 0x002865 (0x3d79a bytes) 000948 1789518962.868549 write 0x005f19 thru 0x00731f (0x1407 bytes) 000949 1789518962.869667 mapread 0x0042a6 thru 0x00731f (0x307a bytes) 000950 1789518962.874740 punch from 0x002ae9 to 0x004ed1 (0x23e9 bytes) 000951 1789518962.882079 trunc from 0x007320 to 0x02bdce (0x24aaf bytes) 000952 1789518962.897186 fallocate 0x014d3c thru 0x0187c5 (0x3a8a bytes) 000953 1789518962.904130 mapread 0x00637b thru 0x01234d (0xbfd3 bytes) 000954 1789518962.929246 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2bdcf): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bdcf): Mirror 4: 0xd3432c92 (stale) Mirror 5: 0xd3432c92 (stale) Mirror 6: 0xd3432c92 (stale) Mirror 7: 0xd3432c92 (stale) Mirror 8: 0xd3432c92 (stale) CRC-32 checksum value for chunk [0, 0x2bdcf): Mirror 1: 0x1b7d55fc Mirror 9: 0x1b7d55fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bdcf: skipped 000955 1789518963.703207 punch from 0x02407a to 0x026a20 (0x29a7 bytes) 000956 1789518963.738177 trunc from 0x02bdcf to 0x0376ce (0xb900 bytes) 000957 1789518963.763512 fallocate 0x009671 thru 0x017c7f (0xe60f bytes) 000958 1789518963.768930 read 0x036667 thru 0x03751d (0xeb7 bytes) 000959 1789518963.778292 punch from 0x024991 to 0x02c448 (0x7ab8 bytes) 000960 1789518963.782891 mapwrite 0x038150 thru 0x03aee8 (0x2d99 bytes) 000961 1789518963.827665 read 0x006758 thru 0x014fb3 (0xe85c bytes) 000962 1789518963.837614 trunc from 0x03aee9 to 0x007d56 (0x33192 bytes) 000963 1789518963.857306 read 0x005055 thru 0x007d56 (0x2d02 bytes) 000964 1789518963.857873 fallocate 0x03e4b6 thru 0x03ffff (0x1b4a bytes) 000965 1789518963.865880 mapwrite 0x00a063 thru 0x01671d (0xc6bb bytes) 000966 1789518963.895379 write 0x00bd40 thru 0x019f12 (0xe1d3 bytes) 000967 1789518963.896958 mapwrite 0x030661 thru 0x03b6c8 (0xb068 bytes) 000968 1789518963.914846 trunc from 0x03a32f to 0x03ffff (0x5cd1 bytes) 000969 1789518963.918959 write 0x0043a1 thru 0x0099f6 (0x5656 bytes) 000970 1789518963.920058 mapwrite 0x03d033 thru 0x03ffff (0x2fcd bytes) 000971 1789518963.943498 punch from 0x03c122 to 0x03ffff (0x3ede bytes) 000972 1789518963.947793 mapread 0x0015dc thru 0x00ed1a (0xd73f bytes) 000973 1789518963.948814 mapread 0x00b5f8 thru 0x0118d4 (0x62dd bytes) 000974 1789518963.952258 MIRROR_RESYNC 7 mirrors 000975 1789518965.232328 trunc from 0x02a526 to 0x03982a (0xf305 bytes) 000976 1789518965.270318 mapwrite 0x00d93c thru 0x01aa2d (0xd0f2 bytes) 000977 1789518965.312413 punch from 0x03204a to 0x033602 (0x15b9 bytes) 000978 1789518965.316374 fallocate 0x026275 thru 0x026c58 (0x9e4 bytes) 000979 1789518965.321092 trunc from 0x040000 to 0x006da8 (0x39257 bytes) 000980 1789518965.345047 read 0x003ed4 thru 0x006da8 (0x2ed5 bytes) 000981 1789518965.358605 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x6da9): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6da9): Mirror 4: 0xaa1cf7ac (stale) Mirror 5: 0xaa1cf7ac (stale) Mirror 6: 0xaa1cf7ac (stale) Mirror 7: 0xaa1cf7ac (stale) Mirror 8: 0xaa1cf7ac (stale) Mirror 9: 0xaa1cf7ac (stale) CRC-32 checksum value for chunk [0, 0x6da9): Mirror 1: 0xaa1cf7ac Mirror 10: 0xaa1cf7ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6da9: skipped 000982 1789518965.996243 mapwrite 0x015fcd thru 0x022fe1 (0xd015 bytes) 000983 1789518966.095222 fallocate 0x008333 thru 0x012b9c (0xa86a bytes) 000986 1789518966.103276 trunc from 0x016eba to 0x022fe1 (0xc128 bytes) 000988 1789518966.117895 MIRROR_RESYNC 8 mirrors 000989 1789518967.585807 MIRROR_EXTEND to 9 mirrors 000990 1789518968.379608 MIRROR_RESYNC 9 mirrors 000991 1789518968.763178 mapread 0x01dec4 thru 0x022fe1 (0x511e bytes) 000992 1789518968.781248 mapread 0x015553 thru 0x01d318 (0x7dc6 bytes) 000993 1789518968.787649 trunc from 0x00e7a3 to 0x016d1d (0x857b bytes) 000994 1789518968.816745 fallocate 0x014c5d thru 0x018b15 (0x3eb9 bytes) 000996 1789518968.825531 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x2a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22fe2): Mirror 4: 0xa9f1dad4 (stale) Mirror 6: 0xa9f1dad4 (stale) Mirror 7: 0xa9f1dad4 (stale) Mirror 8: 0xa9f1dad4 (stale) Mirror 9: 0xa9f1dad4 (stale) Mirror 10: 0xa9f1dad4 (stale) Mirror 11: 0xa9f1dad4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22fe2): Mirror 4: 0xa9f1dad4 (stale) Mirror 6: 0xa9f1dad4 (stale) Mirror 7: 0xa9f1dad4 (stale) Mirror 8: 0xa9f1dad4 (stale) Mirror 9: 0xa9f1dad4 (stale) Mirror 10: 0xa9f1dad4 (stale) Mirror 11: 0xa9f1dad4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22fe2: skipped 000997 1789518969.186923 mapread 0x011e91 thru 0x01f7fd (0xd96d bytes) 000998 1789518969.216170 trunc from 0x008f47 to 0x0159c6 (0xca80 bytes) 000999 1789518969.219682 punch from 0x00ea5d to 0x016db9 (0x835d bytes) 001000 1789518969.226249 write 0x0104da thru 0x01d956 (0xd47d bytes) All operations completed A-OK! PASS 70a (57s) == sanity-flr test 71: check mirror extend parent fid ==== 20:36:13 (1789518973) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x2db:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (6s) == sanity-flr test 72: check append on FLR file ========== 20:36:19 (1789518979) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150247 s, 14.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765266 s, 13.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (6s) == sanity-flr test 200a: stress test ===================== 20:36:25 (1789518985) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre2 Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5839360 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2939072 .. Extending file size to 6849376 .. Extending file size to 5710368 .. Extending file size to 8480160 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8677088 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7787008 .. failed Extending file size to 8846336 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7973696 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8429344 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8703168 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9037632 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8985984 .. Extending file size to 9014976 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 61991 61993 61995 61996 61997 192.168.206.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre2 (opts:) 192.168.206.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (70s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:37:35 (1789519055) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre2 Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4026944 .. Extending file size to 6051488 .. failed Extending file size to 6392736 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7336416 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7383136 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8952512 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8100704 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8537344 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9157664 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9025440 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 63949 63950 63952 63954 63956 failed 192.168.206.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre2 (opts:) 192.168.206.131@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (73s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:38:48 (1789519128) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1268960 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2332 1268972 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18908 3588112 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24696 3582324 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43604 7170436 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2344 1268960 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2332 1268972 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18908 3588112 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24696 3582324 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43604 7170436 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0875818 s, 23.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11815 s, 17.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x39da0dae (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre2 (opts:) PASS 200c (11s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:39:00 (1789519140) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.076722 s, 27.3 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 20:39:05 (1789519145) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } PASS 203 (4s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:39:09 (1789519149) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0709172 s, 29.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3d6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:39:14 (1789519154) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0552605 s, 38.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:39:18 (1789519158) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0987437 s, 21.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } PASS 204c (27s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:39:45 (1789519185) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg631-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308597 s, 34.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3d5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3d5:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3d8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3d6:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:40:28 (1789519228) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg631-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0720903 s, 29.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (41s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:41:09 (1789519269) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg631-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg631-server: Pool lustre.test_204f created Adding targets to pool oleg631-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg631-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0942475 s, 22.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg631-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg631-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg631-server: Pool lustre.test_204f destroyed PASS 204f (63s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:42:12 (1789519332) PASS 205a (4s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:42:16 (1789519336) PASS 205b (4s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:42:21 (1789519341) oleg631-server: Pool lustre.test_206 created Adding targets to pool oleg631-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg631-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg631-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg631-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg631-server: Pool lustre.test_206 destroyed PASS 206 (18s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:42:39 (1789519359) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348643 s, 30.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409881 s, 25.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf00b75da Mirror 3: 0xf00b75da PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:42:46 (1789519366) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:42:48 (1789519368) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:42:50 (1789519370) Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0434572 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x22f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1aa0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1aa0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1aa0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xee0000 Read this (4 bytes): BBBB 192.168.206.131@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg631-client.virtnet /mnt/lustre2 (opts:) PASS 209a (22s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:43:13 (1789519393) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00750856 s, 1.1 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:43:16 (1789519396) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217595 s, 48.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611268 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 20:43:17.000000000 -0400 Modify: 2026-09-15 20:43:17.000000000 -0400 Change: 2026-09-15 20:43:18.000000000 -0400 Birth: 2026-09-15 20:43:16.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:43:20 (1789519400) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172674 s, 60.7 MB/s oleg631-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg631-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg631-server: O/280000401/d30/13e: File not found by ext2_lookup PASS 210b (16s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:43:36 (1789519416) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.143916 s, 285 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0143478 s, 285 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 15 20:43 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x89:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 15 20:43 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:43:40 (1789519420) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:43:44 (1789519424) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 1141 sec =========== 20:43:49 (1789519429) === sanity-flr: start cleanup 20:43:50 (1789519430) === === sanity-flr: finish cleanup 20:43:56 (1789519436) ===