-----============= acceptance-small: sanity-flr ============----- Tue Aug 11 12:03:42 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg405-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:03:50 (1786464230) === oleg405-client.virtnet: executing check_config_client /mnt/lustre oleg405-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg405-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f32c4d04800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f32c4d04800.idle_timeout=debug disable quota as required oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:04:04 (1786464244) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7633] == sanity-flr test 0a: lfs mirror create with -N option == 12:04:05 (1786464245) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:04:12 (1786464252) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:04:13 (1786464253) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:04:15 (1786464255) PASS 0d (9s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:04:24 (1786464264) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:04:26 (1786464266) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 12:04:27 (1786464267) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f32c4d04800.stats=clear osc.lustre-OST0001-osc-ffff9f32c4d04800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:04:31 (1786464271) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:04:35 (1786464275) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:04:38 (1786464278) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:04:40 (1786464280) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:04:43 (1786464283) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 12:04:47 (1786464287) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:04:50 (1786464290) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:04:51 (1786464291) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:04:54 (1786464294) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:04:58 (1786464298) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:05:01 (1786464301) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.601352 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733427 s, 14.3 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:05:06 (1786464306) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.16605 s, 107 MB/s PASS 22 (3s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:05:09 (1786464309) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:05:13 (1786464313) Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f32c4d04800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f32c4d04800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f32c4d04800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f32c4d04800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (29s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:05:42 (1786464342) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0822429 s, 51.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f32c4d04800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f32c4d04800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (14s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:05:56 (1786464356) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f32c4d04800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f32c4d04800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f32c4d04800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f32c4d04800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (30s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:06:26 (1786464386) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53827 s, 34.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 11 12:06 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.333s user 0m0.001s sys 0m0.166s Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f32c4d04800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f32c4d04800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.363s user 0m0.001s sys 0m0.169s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f32c4d04800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f32c4d04800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.433s user 0m0.001s sys 0m0.172s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg405-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f32c4d04800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f32c4d04800.ost_server_uuid in FULL state after 0 sec PASS 33b (34s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:07:00 (1786464420) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:07:01 (1786464421) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:07:03 (1786464423) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 12:07:04 (1786464424) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 12:07:07 (1786464427) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.320424 s, 39.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:07:13 (1786464433) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236165 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156018 s, 67.2 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:07:15 (1786464435) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:07:19 (1786464439) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00098106 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00074135 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00066436 s, 1.5 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000901278 s, 1.1 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 12:07:23 (1786464443) create test files with size 15739384 .. 9671ad4a25028d8bb32d9eaa9f2e6042 - 7cb3df1c2f375e1dcd4ab444d7eb2275 - 367dafd1abb40581e57c413a66db8bba - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre PASS 37 (7s) == sanity-flr test 38: resync ============================ 12:07:31 (1786464451) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (5s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:07:36 (1786464456) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:07:39 (1786464459) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04505 s, 23.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132731 s, 79.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 12:07:42 (1786464462) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0429803 s, 48.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.131731 s, 31.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0869312 s, 24.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 12:07:48 (1786464468) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 12:07:49 (1786464469) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:07:51 (1786464471) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160319 s, 65.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 12:07:54 (1786464474) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:07:55 (1786464475) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:07:59 (1786464479) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.105635 s, 99.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:08:16 (1786464496) llite.lustre-ffff9f32c63f1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0993643 s, 106 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9f32c8e43800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:08:21 (1786464501) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (14s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:08:35 (1786464515) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:08:49 (1786464529) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 12:08:50 (1786464530) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0745914 s, 42.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0665272 s, 47.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xbb529294 (nosync,stale) Mirror 2: 0xbb529294 (nosync,stale) ** verify mirror 2 doesn't change original checksum: a9034146416034693a484dc1d2c371df - mirror 1 checksum: 7f4be3b2b41fd215dbef217b0d3a8a81 - mirror 2 checksum: a9034146416034693a484dc1d2c371df - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-11T12:08:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-11T12:08:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a9034146416034693a484dc1d2c371df - mirror 1 checksum: 7f4be3b2b41fd215dbef217b0d3a8a81 - mirror 2 checksum: 7f4be3b2b41fd215dbef217b0d3a8a81 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-11T12:08:52-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-11T12:08:52-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:08:54 (1786464534) Failing mds1 on oleg405-server Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 12:08:57 (1786464537) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server Failover mds1 to oleg405-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 12:09:12 (1786464552) targets are mounted 12:09:12 (1786464552) facet_failover done oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128323 s, 81.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (25s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:09:19 (1786464559) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:09:20 (1786464560) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:09:21 (1786464561) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00321149 s, 5.1 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:09:24 (1786464564) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:09:25 (1786464565) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00215487 s, 1.9 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0228597 s, 179 kB/s PASS 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:09:28 (1786464568) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:09:35 (1786464575) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:09:56 (1786464596) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:10:14 (1786464614) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:10:32 (1786464632) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 12:10:35 (1786464635) mirror create pid 54250 mirror split pid 54251 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7817: 54251 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54250 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 12:11:41 (1786464701) Seed set to 4702 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x2b533 000002 1786464702.755424 trunc from 00000000 to 0x02b532 (0x2b533 bytes) 000006 1786464702.779947 write 0x00616e thru 0x006b51 (0x9e4 bytes) 000008 1786464702.788693 read 0x028163 thru 0x02b532 (0x33d0 bytes) 000009 1786464702.796619 write 0x01f0e2 thru 0x020e4c (0x1d6b bytes) 000012 1786464702.815306 mapwrite 0x030da7 thru 0x033cf7 (0x2f51 bytes) 000014 1786464702.899677 mapread 0x02ea51 thru 0x031c0d (0x31bd bytes) 000015 1786464702.901408 mapread 0x025001 thru 0x027d01 (0x2d01 bytes) 000018 1786464702.913681 write 0x038b93 thru 0x03ffff (0x746d bytes) 000019 1786464702.915523 mapwrite 0x006e4d thru 0x009b33 (0x2ce7 bytes) 000020 1786464702.962692 mapread 0x021485 thru 0x02fc78 (0xe7f4 bytes) 000023 1786464702.965784 write 0x00077b thru 0x006bf7 (0x647d bytes) 000025 1786464702.971251 mapread 0x00dd39 thru 0x00f67f (0x1947 bytes) 000027 1786464702.973944 mapread 0x036efd thru 0x03ffff (0x9103 bytes) 000028 1786464702.980999 mapwrite 0x00f6c1 thru 0x01f53c (0xfe7c bytes) 000029 1786464703.049382 read 0x004fa7 thru 0x0130c6 (0xe120 bytes) 000030 1786464703.050755 trunc from 0x040000 to 0x0129c3 (0x2d63c bytes) 000035 1786464703.076985 MIRROR_EXTEND to 2 mirrors 000036 1786464703.570947 read 0x0124bb thru 0x0129c3 (0x509 bytes) 000039 1786464703.602754 trunc from 0x0129c4 to 0x00e425 (0x459e bytes) 000045 1786464703.669868 mapwrite 0x00a4ef thru 0x015a7b (0xb58d bytes) 000046 1786464703.763713 read 0x0101c2 thru 0x01563f (0x547e bytes) 000047 1786464703.764764 mapread 0x0147f9 thru 0x015a7b (0x1283 bytes) 000049 1786464703.777159 MIRROR_RESYNC 2 mirrors 000050 1786464704.257123 mapwrite 0x02161f thru 0x02e59f (0xcf81 bytes) 000052 1786464704.375117 MIRROR_RESYNC 2 mirrors 000054 1786464704.926420 mapread 0x01280a thru 0x019fb8 (0x77af bytes) 000056 1786464704.948545 read 0x010738 thru 0x011252 (0xb1b bytes) 000057 1786464704.949460 mapread 0x01c54c thru 0x024cb9 (0x876e bytes) 000059 1786464704.961258 write 0x00db04 thru 0x015e19 (0x8316 bytes) 000062 1786464705.024431 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000064 1786464705.132694 mapread 0x027b11 thru 0x02e59f (0x6a8f bytes) 000065 1786464705.152360 trunc from 0x02e5a0 to 0x01bd94 (0x1280b bytes) 000067 1786464705.186329 MIRROR_EXTEND to 2 mirrors 000068 1786464705.783303 read 0x00aca4 thru 0x00f9fe (0x4d5b bytes) 000069 1786464705.804586 MIRROR_RESYNC 2 mirrors 000070 1786464706.039244 mapread 0x00bd95 thru 0x015597 (0x9803 bytes) 000071 1786464706.083889 MIRROR_RESYNC 2 mirrors 000072 1786464706.322994 MIRROR_EXTEND to 3 mirrors 000074 1786464706.748260 write 0x022013 thru 0x0229c2 (0x9b0 bytes) 000075 1786464706.804697 mapwrite 0x003234 thru 0x00b36d (0x813a bytes) 000077 1786464706.863703 trunc from 0x0229c3 to 0x0173b5 (0xb60d bytes) 000079 1786464706.879514 mapread 0x004c2e thru 0x007828 (0x2bfb bytes) 000080 1786464706.880598 write 0x03b6d9 thru 0x03ffff (0x4927 bytes) truncating to largest ever: 0x3f9d2 000081 1786464706.881913 trunc from 0x040000 to 0x03f9d1 (0x62e bytes) 000082 1786464706.897142 write 0x03febf thru 0x03ffff (0x141 bytes) 000084 1786464706.898572 mapwrite 0x032d20 thru 0x03ffff (0xd2e0 bytes) 000085 1786464706.936003 trunc from 0x040000 to 0x03c078 (0x3f87 bytes) 000088 1786464706.950315 mapread 0x010eb3 thru 0x01fba7 (0xecf5 bytes) 000089 1786464706.964324 read 0x01ad38 thru 0x027e98 (0xd161 bytes) 000090 1786464706.970804 read 0x013c53 thru 0x01edee (0xb19c bytes) 000093 1786464706.972474 read 0x000660 thru 0x010096 (0xfa37 bytes) 000098 1786464706.983517 mapread 0x023f6f thru 0x027f74 (0x4006 bytes) 000102 1786464706.989458 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c079): Mirror 2: 0x54e020af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c079): Mirror 2: 0x54e020af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c079: skipped 000103 1786464707.179214 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000104 1786464707.268795 read 0x00cbac thru 0x019333 (0xc788 bytes) 000105 1786464707.277514 read 0x024f9a thru 0x02f726 (0xa78d bytes) 000106 1786464707.286034 trunc from 0x03c079 to 0x01d1e1 (0x1ee97 bytes) 000108 1786464707.300648 write 0x0045de thru 0x011d4d (0xd770 bytes) 000110 1786464707.302904 read 0x000057 thru 0x0036f7 (0x36a1 bytes) 000115 1786464707.307594 mapwrite 0x003fce thru 0x011132 (0xd165 bytes) 000116 1786464707.342057 read 0x013d61 thru 0x01d1e1 (0x9481 bytes) 000117 1786464707.343222 mapread 0x0171cd thru 0x01cee3 (0x5d17 bytes) 000120 1786464707.347230 trunc from 0x01d1e2 to 0x03a794 (0x1d5b3 bytes) 000121 1786464707.360869 read 0x021447 thru 0x0229e5 (0x159f bytes) 000122 1786464707.368311 trunc from 0x03a795 to 0x031c90 (0x8b04 bytes) 000123 1786464707.381735 mapread 0x011f8c thru 0x012981 (0x9f6 bytes) 000124 1786464707.383087 write 0x011518 thru 0x01a0d6 (0x8bbf bytes) 000127 1786464707.385763 write 0x0330cd thru 0x03d184 (0xa0b8 bytes) 000131 1786464707.389952 trunc from 0x03d185 to 0x02796d (0x15817 bytes) 000132 1786464707.403952 mapread 0x0275de thru 0x02796d (0x390 bytes) 000133 1786464707.405191 read 0x003f75 thru 0x00c304 (0x8390 bytes) 000135 1786464707.406597 write 0x03a85f thru 0x03ffff (0x57a1 bytes) 000136 1786464707.407957 mapwrite 0x00b1aa thru 0x0179db (0xc832 bytes) 000138 1786464707.460911 MIRROR_EXTEND to 2 mirrors 000139 1786464707.969469 mapread 0x03d06b thru 0x03d142 (0xd8 bytes) 000143 1786464708.014506 write 0x00ef1a thru 0x01a776 (0xb85d bytes) 000144 1786464708.085047 mapwrite 0x025da4 thru 0x027674 (0x18d1 bytes) 000145 1786464708.135455 write 0x021f69 thru 0x027d94 (0x5e2c bytes) 000148 1786464708.138586 trunc from 0x040000 to 0x038537 (0x7ac8 bytes) 000151 1786464708.176252 MIRROR_RESYNC 2 mirrors 000152 1786464708.772462 MIRROR_RESYNC 2 mirrors 000153 1786464709.024244 MIRROR_RESYNC 2 mirrors 000154 1786464709.210888 mapread 0x0366e4 thru 0x038537 (0x1e54 bytes) 000155 1786464709.231414 MIRROR_EXTEND to 3 mirrors 000157 1786464709.679146 trunc from 0x038538 to 0x004b77 (0x339c0 bytes) 000159 1786464709.730799 read 0x001f55 thru 0x004b77 (0x2c23 bytes) 000160 1786464709.735418 mapwrite 0x0053b9 thru 0x013fb8 (0xec00 bytes) 000162 1786464709.802545 read 0x00caf1 thru 0x013fb8 (0x74c8 bytes) 000163 1786464709.806545 MIRROR_RESYNC 3 mirrors 000165 1786464710.246846 write 0x02f289 thru 0x032a7a (0x37f2 bytes) 000166 1786464710.291180 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32a7b): Mirror 3: 0x1cf165b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a7b): Mirror 3: 0x1cf165b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a7b: skipped 000167 1786464710.481545 write 0x02abf8 thru 0x0341e9 (0x95f2 bytes) 000168 1786464710.486288 mapwrite 0x005c08 thru 0x0157ba (0xfbb3 bytes) 000170 1786464710.543220 read 0x019978 thru 0x01e1dd (0x4866 bytes) 000171 1786464710.548898 trunc from 0x0341ea to 0x011596 (0x22c53 bytes) 000172 1786464710.559129 mapread 0x009ec6 thru 0x011596 (0x76d1 bytes) 000173 1786464710.560371 write 0x02a1ff thru 0x02b754 (0x1556 bytes) 000174 1786464710.561520 trunc from 0x02b755 to 0x00a38b (0x213c9 bytes) 000177 1786464710.575503 write 0x037685 thru 0x03c21d (0x4b99 bytes) 000178 1786464710.577209 mapwrite 0x03174d thru 0x032386 (0xc3a bytes) 000179 1786464710.624723 trunc from 0x03c21e to 0x028ea5 (0x13378 bytes) 000181 1786464710.645217 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000182 1786464710.750558 read 0x00603b thru 0x00d5a6 (0x756c bytes) 000183 1786464710.757178 mapread 0x00e2b7 thru 0x01a87f (0xc5c9 bytes) 000185 1786464710.780888 read 0x022980 thru 0x0253cf (0x2a50 bytes) 000190 1786464710.789931 mapwrite 0x003ef6 thru 0x010e00 (0xcf0b bytes) 000191 1786464710.825656 trunc from 0x028ea6 to 0x0029b5 (0x264f0 bytes) 000192 1786464710.838759 write 0x03b5fb thru 0x03ffff (0x4a05 bytes) 000193 1786464710.840138 mapwrite 0x0154a9 thru 0x021361 (0xbeb9 bytes) 000199 1786464710.946194 mapwrite 0x03b2a9 thru 0x03d137 (0x1e8f bytes) 000200 1786464710.974599 write 0x037a16 thru 0x03ffff (0x85ea bytes) 000201 1786464710.977235 trunc from 0x040000 to 0x02ed71 (0x1128e bytes) 000203 1786464710.989868 write 0x037906 thru 0x0397dc (0x1ed7 bytes) 000206 1786464710.999186 mapwrite 0x02b2e7 thru 0x035816 (0xa530 bytes) 000210 1786464711.068164 mapwrite 0x032405 thru 0x03ee29 (0xca25 bytes) 000212 1786464711.149161 mapread 0x006349 thru 0x0066a5 (0x35d bytes) 000213 1786464711.153098 write 0x026b09 thru 0x02897f (0x1e77 bytes) 000216 1786464711.163194 mapread 0x031f1b thru 0x033aba (0x1ba0 bytes) 000219 1786464711.168377 MIRROR_EXTEND to 2 mirrors 000221 1786464711.675834 mapwrite 0x0172fb thru 0x019e43 (0x2b49 bytes) 000222 1786464711.834194 read 0x0079f3 thru 0x0080ec (0x6fa bytes) 000224 1786464711.845162 MIRROR_RESYNC 2 mirrors 000225 1786464712.313254 mapwrite 0x02660a thru 0x032c55 (0xc64c bytes) 000227 1786464712.413940 mapwrite 0x004ab5 thru 0x01354d (0xea99 bytes) 000228 1786464712.466926 trunc from 0x03ee2a to 0x02ede1 (0x10048 bytes) 000229 1786464712.480268 read 0x020e86 thru 0x02d12c (0xc2a7 bytes) 000232 1786464712.482132 mapwrite 0x03890f thru 0x03fbc7 (0x72b9 bytes) 000233 1786464712.533595 read 0x006c81 thru 0x00d164 (0x64e4 bytes) 000234 1786464712.534312 write 0x007e03 thru 0x0167d6 (0xe9d4 bytes) 000237 1786464712.539105 mapread 0x0341a0 thru 0x03fbc7 (0xba28 bytes) 000238 1786464712.545510 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000239 1786464712.663107 mapread 0x00316e thru 0x00a042 (0x6ed5 bytes) 000241 1786464712.692598 mapread 0x028df0 thru 0x02c6f6 (0x3907 bytes) 000243 1786464712.707681 read 0x00c41d thru 0x00ffc5 (0x3ba9 bytes) 000245 1786464712.709289 trunc from 0x03fbc8 to 0x03d618 (0x25af bytes) 000247 1786464712.726513 write 0x037c4c thru 0x03a775 (0x2b2a bytes) 000248 1786464712.739640 mapwrite 0x02a8e7 thru 0x030f42 (0x665c bytes) 000249 1786464712.801273 trunc from 0x03d619 to 0x0254ed (0x1812b bytes) 000252 1786464712.820866 mapread 0x005039 thru 0x00a800 (0x57c8 bytes) 000254 1786464712.822958 trunc from 0x0254ee to 0x02fbca (0xa6dd bytes) 000255 1786464712.839986 MIRROR_EXTEND to 2 mirrors 000256 1786464713.358574 write 0x0083d9 thru 0x00d0ee (0x4d16 bytes) 000257 1786464713.422445 mapread 0x0268b7 thru 0x02dfb3 (0x76fd bytes) 000258 1786464713.445523 mapread 0x02bc68 thru 0x02cdb9 (0x1152 bytes) 000260 1786464713.448914 read 0x00fc62 thru 0x01b5ed (0xb98c bytes) 000261 1786464713.459213 read 0x02b066 thru 0x02fbca (0x4b65 bytes) 000263 1786464713.464806 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000264 1786464713.593811 MIRROR_EXTEND to 2 mirrors 000265 1786464714.120484 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000267 1786464714.211508 mapwrite 0x0119d3 thru 0x012f45 (0x1573 bytes) 000270 1786464714.260519 MIRROR_EXTEND to 2 mirrors 000271 1786464714.758189 mapwrite 0x0309ae thru 0x0357bd (0x4e10 bytes) 000274 1786464714.873789 mapread 0x027912 thru 0x02e15e (0x684d bytes) 000275 1786464714.883466 trunc from 0x0357be to 0x004497 (0x31326 bytes) 000276 1786464714.900861 mapread 0x003b9c thru 0x004497 (0x8fc bytes) 000281 1786464714.909431 trunc from 0x004498 to 0x01f5e0 (0x1b149 bytes) 000282 1786464714.922546 read 0x0163e0 thru 0x017056 (0xc77 bytes) 000283 1786464714.927136 read 0x012a32 thru 0x01b465 (0x8a34 bytes) 000285 1786464714.936659 trunc from 0x01f5e1 to 0x01cf7a (0x2666 bytes) 000286 1786464714.948886 mapwrite 0x01beb7 thru 0x02205a (0x61a4 bytes) 000287 1786464715.021346 MIRROR_RESYNC 2 mirrors 000288 1786464715.478611 trunc from 0x02205b to 0x021114 (0xf46 bytes) 000289 1786464715.535195 read 0x0109d7 thru 0x01adcc (0xa3f6 bytes) 000290 1786464715.542354 write 0x0313d8 thru 0x03ffff (0xec28 bytes) 000291 1786464715.544592 write 0x0143db thru 0x022629 (0xe24f bytes) 000292 1786464715.553884 MIRROR_RESYNC 2 mirrors 000294 1786464716.133403 trunc from 0x040000 to 0x006557 (0x39aa8 bytes) 000296 1786464716.217360 mapread 0x006311 thru 0x006557 (0x247 bytes) 000297 1786464716.224662 write 0x009fbe thru 0x019495 (0xf4d8 bytes) 000298 1786464716.227395 write 0x0322f6 thru 0x038517 (0x6222 bytes) 000299 1786464716.229120 write 0x026017 thru 0x02a425 (0x440f bytes) 000301 1786464716.240940 mapwrite 0x0081e8 thru 0x0090da (0xef3 bytes) 000302 1786464716.277382 mapread 0x016c76 thru 0x021eb5 (0xb240 bytes) 000303 1786464716.287395 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38518): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38518): Mirror 2: 0xcf37a179 (stale) CRC-32 checksum value for chunk [0, 0x38518): Mirror 1: 0x276cdb39 Mirror 3: 0x276cdb39 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38518: skipped 000304 1786464716.945888 read 0x00ef98 thru 0x016145 (0x71ae bytes) 000305 1786464716.962470 write 0x00dfa6 thru 0x014984 (0x69df bytes) 000308 1786464717.029512 trunc from 0x038518 to 0x01e679 (0x19e9e bytes) 000309 1786464717.066616 write 0x0363ff thru 0x0371f9 (0xdfb bytes) 000310 1786464717.068264 trunc from 0x0371fa to 0x01775a (0x1fa9f bytes) 000313 1786464717.086252 read 0x017246 thru 0x01775a (0x515 bytes) 000314 1786464717.097151 read 0x00b3aa thru 0x01775a (0xc3b1 bytes) 000315 1786464717.097878 write 0x03a817 thru 0x03ea13 (0x41fd bytes) 000317 1786464717.099773 mapread 0x00aeca thru 0x0128ee (0x7a25 bytes) 000319 1786464717.101693 mapwrite 0x015a4d thru 0x01b85b (0x5e0f bytes) 000323 1786464717.153970 read 0x03639d thru 0x03ea13 (0x8677 bytes) 000324 1786464717.160287 MIRROR_RESYNC 3 mirrors 000325 1786464717.777271 write 0x008675 thru 0x014ad1 (0xc45d bytes) 000329 1786464717.849224 write 0x012d2f thru 0x013f00 (0x11d2 bytes) 000330 1786464717.852058 read 0x0364e6 thru 0x03dcb1 (0x77cc bytes) 000331 1786464717.859217 mapwrite 0x004054 thru 0x00a0a4 (0x6051 bytes) 000332 1786464717.913495 trunc from 0x03ea14 to 0x03cf40 (0x1ad3 bytes) 000335 1786464717.947608 mapwrite 0x035316 thru 0x03ec28 (0x9913 bytes) 000336 1786464718.019086 write 0x03837d thru 0x03ffff (0x7c83 bytes) 000337 1786464718.020491 mapwrite 0x018354 thru 0x027874 (0xf521 bytes) 000338 1786464718.078924 mapread 0x0139f1 thru 0x01f7ca (0xbdda bytes) 000339 1786464718.080324 trunc from 0x040000 to 0x026162 (0x19e9d bytes) 000340 1786464718.095529 MIRROR_RESYNC 3 mirrors 000345 1786464718.615992 trunc from 0x026163 to 0x02b684 (0x5522 bytes) 000347 1786464718.675372 write 0x0322af thru 0x03d1da (0xaf2c bytes) 000348 1786464718.677030 read 0x00a4af thru 0x013e7e (0x99d0 bytes) 000349 1786464718.682576 mapwrite 0x00c2b3 thru 0x010ae5 (0x4833 bytes) 000350 1786464718.717598 write 0x014d77 thru 0x023bc4 (0xee4e bytes) 000353 1786464718.728293 read 0x0073c0 thru 0x00aa8a (0x36cb bytes) 000354 1786464718.736900 mapwrite 0x013c57 thru 0x015229 (0x15d3 bytes) 000355 1786464718.773167 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d1db): Mirror 3: 0xc22aa59 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1db): Mirror 3: 0xc22aa59 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1db: skipped 000356 1786464718.965618 read 0x033a0b thru 0x03432c (0x922 bytes) 000357 1786464718.970591 mapread 0x02b8b5 thru 0x03b50d (0xfc59 bytes) 000358 1786464718.999599 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000362 1786464719.109645 MIRROR_EXTEND to 2 mirrors 000363 1786464719.659561 read 0x00db07 thru 0x0131ba (0x56b4 bytes) 000364 1786464719.671119 mapread 0x01ce99 thru 0x028e4c (0xbfb4 bytes) 000367 1786464719.734769 write 0x020dcb thru 0x0307a4 (0xf9da bytes) 000369 1786464719.801381 mapwrite 0x028bb5 thru 0x0355ad (0xc9f9 bytes) 000370 1786464719.865580 mapwrite 0x004663 thru 0x00946a (0x4e08 bytes) 000372 1786464719.919870 trunc from 0x03d1db to 0x02116a (0x1c070 bytes) 000376 1786464719.940375 mapread 0x01348d thru 0x01fe04 (0xc978 bytes) 000377 1786464719.955855 write 0x03962a thru 0x03ffff (0x69d6 bytes) 000378 1786464719.957642 mapread 0x023ea1 thru 0x02854c (0x46ac bytes) 000379 1786464719.965427 trunc from 0x040000 to 0x01f6d1 (0x2092e bytes) 000381 1786464719.982398 trunc from 0x01f6d2 to 0x03f252 (0x1fb81 bytes) 000383 1786464720.007544 trunc from 0x03f253 to 0x01193c (0x2d916 bytes) 000386 1786464720.027163 mapread 0x00e032 thru 0x01193c (0x390b bytes) 000387 1786464720.028568 mapwrite 0x000728 thru 0x00188c (0x1165 bytes) 000388 1786464720.084828 mapwrite 0x0010a7 thru 0x00b9c6 (0xa920 bytes) 000389 1786464720.127509 mapread 0x0034ed thru 0x0116cd (0xe1e1 bytes) 000390 1786464720.133862 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000391 1786464720.269972 mapwrite 0x03d6ad thru 0x03fcb1 (0x2605 bytes) 000392 1786464720.355051 trunc from 0x03fcb2 to 0x007c3b (0x38076 bytes) 000394 1786464720.375624 read 0x0045b1 thru 0x007c3b (0x368b bytes) 000395 1786464720.392342 MIRROR_EXTEND to 2 mirrors 000396 1786464720.837400 trunc from 0x007c3c to 0x0298e4 (0x21ca9 bytes) 000398 1786464720.949296 MIRROR_RESYNC 2 mirrors 000400 1786464721.480772 mapread 0x024c16 thru 0x0277f4 (0x2bdf bytes) 000401 1786464721.498909 mapread 0x003a66 thru 0x00d8b5 (0x9e50 bytes) 000402 1786464721.507968 write 0x03077e thru 0x031931 (0x11b4 bytes) 000403 1786464721.553407 mapwrite 0x02185c thru 0x02f1ee (0xd993 bytes) 000404 1786464721.617250 mapwrite 0x032a8f thru 0x037f67 (0x54d9 bytes) 000406 1786464721.711253 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37f68): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f68): Mirror 2: 0xf2c7669b (stale) CRC-32 checksum value for chunk [0, 0x37f68): Mirror 1: 0x62d72bfa Mirror 3: 0x62d72bfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f68: skipped 000407 1786464722.468934 write 0x009e52 thru 0x00a7a4 (0x953 bytes) 000410 1786464722.546206 write 0x00ea10 thru 0x013108 (0x46f9 bytes) 000419 1786464722.559668 mapwrite 0x02ea43 thru 0x02faf1 (0x10af bytes) 000420 1786464722.601102 read 0x037e46 thru 0x037f67 (0x122 bytes) 000421 1786464722.605202 write 0x029f56 thru 0x0317bf (0x786a bytes) 000422 1786464722.614957 trunc from 0x037f68 to 0x00ac4a (0x2d31d bytes) 000423 1786464722.635789 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xac4b): Mirror 3: 0x6fc11fd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xac4b): Mirror 3: 0x6fc11fd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xac4b: skipped 000424 1786464722.833792 trunc from 0x00ac4b to 0x003950 (0x72fa bytes) 000425 1786464722.846813 write 0x0276d8 thru 0x0316ca (0x9ff3 bytes) 000426 1786464722.849785 mapread 0x01058b thru 0x01cf04 (0xc97a bytes) 000427 1786464722.870254 trunc from 0x0316cb to 0x01cd7a (0x14950 bytes) 000428 1786464722.885989 write 0x01e0ed thru 0x01ebfa (0xb0e bytes) 000429 1786464722.887949 write 0x026462 thru 0x027018 (0xbb7 bytes) 000431 1786464722.890375 trunc from 0x027019 to 0x0339a5 (0xc98d bytes) 000433 1786464722.905884 mapwrite 0x005fbd thru 0x00c786 (0x67ca bytes) 000434 1786464722.961330 mapwrite 0x01db86 thru 0x02704a (0x94c5 bytes) 000436 1786464723.009047 write 0x0207c6 thru 0x0301d8 (0xfa13 bytes) 000437 1786464723.013960 mapread 0x0134b0 thru 0x0205d2 (0xd123 bytes) 000440 1786464723.016711 read 0x02b42d thru 0x02fbc8 (0x479c bytes) 000441 1786464723.017715 read 0x01d9bf thru 0x021840 (0x3e82 bytes) 000442 1786464723.018486 mapwrite 0x023690 thru 0x02877f (0x50f0 bytes) 000443 1786464723.057449 trunc from 0x0339a6 to 0x014e5a (0x1eb4b bytes) 000444 1786464723.072611 read 0x0078ae thru 0x00e622 (0x6d75 bytes) 000446 1786464723.074721 read 0x00e37d thru 0x014e5a (0x6ade bytes) 000447 1786464723.079293 MIRROR_RESYNC 2 mirrors 000450 1786464723.576788 write 0x032dfc thru 0x03a510 (0x7715 bytes) 000451 1786464723.641839 write 0x00a5f0 thru 0x00d463 (0x2e74 bytes) 000452 1786464723.652411 mapread 0x014d20 thru 0x020a03 (0xbce4 bytes) 000453 1786464723.687615 write 0x03a498 thru 0x03ffff (0x5b68 bytes) 000454 1786464723.691039 read 0x03f59d thru 0x03ffff (0xa63 bytes) 000455 1786464723.693088 mapread 0x0259a0 thru 0x02f7ef (0x9e50 bytes) 000456 1786464723.703623 read 0x038bac thru 0x03e5f8 (0x5a4d bytes) 000458 1786464723.705511 mapread 0x036cd2 thru 0x039c7e (0x2fad bytes) 000459 1786464723.706508 mapread 0x01ee14 thru 0x025f5d (0x714a bytes) 000460 1786464723.707634 trunc from 0x040000 to 0x00732a (0x38cd5 bytes) 000462 1786464723.741911 trunc from 0x00732b to 0x02a795 (0x2346b bytes) 000463 1786464723.751488 mapread 0x00e803 thru 0x01812d (0x992b bytes) 000464 1786464723.761641 mapread 0x003c04 thru 0x009a85 (0x5e82 bytes) 000467 1786464723.764630 read 0x004888 thru 0x013d72 (0xf4eb bytes) 000468 1786464723.765883 trunc from 0x02a796 to 0x03dd51 (0x135bc bytes) 000474 1786464723.794246 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3dd52): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd52): Mirror 3: 0x540856f8 (stale) CRC-32 checksum value for chunk [0, 0x3dd52): Mirror 1: 0x62ee73e6 Mirror 4: 0x62ee73e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd52: skipped 000475 1786464724.498948 read 0x012010 thru 0x01d2a0 (0xb291 bytes) 000476 1786464724.510130 MIRROR_RESYNC 3 mirrors 000477 1786464724.900386 mapread 0x028eaf thru 0x037799 (0xe8eb bytes) 000478 1786464724.922056 write 0x02a690 thru 0x02e643 (0x3fb4 bytes) 000479 1786464724.964825 write 0x016279 thru 0x01df06 (0x7c8e bytes) 000480 1786464724.978581 MIRROR_RESYNC 3 mirrors 000482 1786464725.786130 mapread 0x03974a thru 0x03a5dc (0xe93 bytes) 000483 1786464725.796963 mapwrite 0x01af47 thru 0x0281c2 (0xd27c bytes) 000484 1786464725.935412 mapread 0x032adc thru 0x03dd51 (0xb276 bytes) 000486 1786464725.952678 mapread 0x021a07 thru 0x023902 (0x1efc bytes) 000487 1786464725.957278 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dd52): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd52): Mirror 3: 0x6df1464e (stale) Mirror 4: 0x6df1464e (stale) CRC-32 checksum value for chunk [0, 0x3dd52): Mirror 1: 0xa1d2fed1 Mirror 5: 0xa1d2fed1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd52: skipped 000488 1786464726.695585 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dd52): Mirror 3: 0x6df1464e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd52): Mirror 3: 0x6df1464e (stale) CRC-32 checksum value for chunk [0, 0x3dd52): Mirror 1: 0xa1d2fed1 Mirror 5: 0xa1d2fed1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd52: skipped 000490 1786464727.096807 trunc from 0x03dd52 to 0x025a6b (0x182e6 bytes) 000492 1786464727.161599 read 0x01a37e thru 0x025a6b (0xb6ee bytes) 000495 1786464727.171927 mapwrite 0x03e8ff thru 0x03ffff (0x1701 bytes) 000496 1786464727.234198 read 0x01e025 thru 0x0249f7 (0x69d3 bytes) 000497 1786464727.234947 write 0x03e282 thru 0x03ffff (0x1d7e bytes) 000501 1786464727.241500 MIRROR_RESYNC 3 mirrors 000502 1786464728.005003 mapwrite 0x03c807 thru 0x03e21b (0x1a15 bytes) 000503 1786464728.116665 mapread 0x00678c thru 0x0139ae (0xd223 bytes) 000506 1786464728.137163 mapread 0x0262ed thru 0x026cb6 (0x9ca bytes) 000507 1786464728.146978 mapwrite 0x03c9f1 thru 0x03ffff (0x360f bytes) 000508 1786464728.178924 MIRROR_RESYNC 3 mirrors 000509 1786464728.905274 mapread 0x015809 thru 0x016185 (0x97d bytes) 000510 1786464728.914126 mapread 0x0134e0 thru 0x0198c3 (0x63e4 bytes) 000511 1786464728.921239 mapwrite 0x011d34 thru 0x0167f8 (0x4ac5 bytes) 000513 1786464729.027871 read 0x026554 thru 0x02816a (0x1c17 bytes) 000514 1786464729.031974 trunc from 0x040000 to 0x034880 (0xb77f bytes) 000516 1786464729.065766 mapread 0x02ee29 thru 0x034880 (0x5a58 bytes) 000517 1786464729.074737 mapread 0x02aee3 thru 0x034880 (0x999e bytes) 000518 1786464729.076236 trunc from 0x034881 to 0x018560 (0x1c320 bytes) 000519 1786464729.090606 read 0x00ca76 thru 0x01701b (0xa5a6 bytes) 000520 1786464729.091428 trunc from 0x018561 to 0x0369e2 (0x1e482 bytes) 000521 1786464729.108516 write 0x02a7d2 thru 0x02fad9 (0x5308 bytes) 000522 1786464729.116438 mapwrite 0x01edc9 thru 0x024228 (0x5460 bytes) 000523 1786464729.155714 trunc from 0x0369e3 to 0x00ea32 (0x27fb0 bytes) 000524 1786464729.175453 write 0x03f4db thru 0x03ffff (0xb25 bytes) 000527 1786464729.179724 write 0x01b3a4 thru 0x01f093 (0x3cf0 bytes) 000529 1786464729.192057 write 0x008a55 thru 0x015c4a (0xd1f6 bytes) 000533 1786464729.200853 read 0x012855 thru 0x017091 (0x483d bytes) 000534 1786464729.208137 mapwrite 0x013d42 thru 0x01b171 (0x7430 bytes) 000535 1786464729.245788 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xeb07e004 (stale) Mirror 5: 0xeb07e004 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96bf9f2d Mirror 6: 0x96bf9f2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000537 1786464730.049802 read 0x003159 thru 0x006ce0 (0x3b88 bytes) 000538 1786464730.064982 write 0x00107c thru 0x008b9f (0x7b24 bytes) 000539 1786464730.136669 write 0x004d80 thru 0x00f0fd (0xa37e bytes) 000543 1786464730.149605 MIRROR_RESYNC 4 mirrors 000547 1786464730.989912 write 0x0066ef thru 0x0069b3 (0x2c5 bytes) 000549 1786464731.041079 MIRROR_RESYNC 4 mirrors 000550 1786464731.864906 mapread 0x025cea thru 0x02f643 (0x995a bytes) 000553 1786464731.880158 trunc from 0x040000 to 0x035948 (0xa6b7 bytes) 000554 1786464731.935198 write 0x011c4f thru 0x017f04 (0x62b6 bytes) 000555 1786464731.945463 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35949): Mirror 5: 0x19519f47 (stale) Mirror 6: 0x19519f47 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35949): Mirror 5: 0x19519f47 (stale) Mirror 6: 0x19519f47 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35949: skipped 000556 1786464732.193963 read 0x011995 thru 0x018c67 (0x72d3 bytes) 000557 1786464732.204706 trunc from 0x035949 to 0x01b0c1 (0x1a887 bytes) 000558 1786464732.227712 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b0c2): Mirror 6: 0x26c8c055 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b0c2): Mirror 6: 0x26c8c055 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b0c2: skipped 000560 1786464732.456210 mapread 0x0186e7 thru 0x01a269 (0x1b83 bytes) 000562 1786464732.474060 mapwrite 0x01d7a5 thru 0x02b187 (0xd9e3 bytes) 000563 1786464732.560922 mapread 0x01a828 thru 0x01f4ec (0x4cc5 bytes) 000564 1786464732.562940 write 0x01a62d thru 0x01d8b6 (0x328a bytes) 000565 1786464732.565826 write 0x011b01 thru 0x01a8d1 (0x8dd1 bytes) 000569 1786464732.580838 read 0x001a14 thru 0x00bae7 (0xa0d4 bytes) 000570 1786464732.589960 read 0x00c37d thru 0x0120fe (0x5d82 bytes) 000571 1786464732.597586 read 0x0192ca thru 0x02629c (0xcfd3 bytes) 000572 1786464732.600235 trunc from 0x02b188 to 0x021217 (0x9f70 bytes) 000574 1786464732.614601 mapwrite 0x035a6d thru 0x036ae2 (0x1076 bytes) 000575 1786464732.685955 read 0x03328f thru 0x036ae2 (0x3854 bytes) 000576 1786464732.686749 mapwrite 0x02f4bd thru 0x03b559 (0xc09d bytes) 000579 1786464732.756064 read 0x03a5fa thru 0x03b559 (0xf60 bytes) 000581 1786464732.758973 write 0x01b352 thru 0x01cace (0x177d bytes) 000582 1786464732.761704 read 0x00a598 thru 0x00cf65 (0x29ce bytes) 000583 1786464732.762743 read 0x014b08 thru 0x019b31 (0x502a bytes) 000584 1786464732.763626 read 0x038e47 thru 0x03b559 (0x2713 bytes) 000585 1786464732.764636 mapwrite 0x012a31 thru 0x018ce5 (0x62b5 bytes) 000586 1786464732.799100 write 0x004bb9 thru 0x00b33d (0x6785 bytes) 000589 1786464732.802445 trunc from 0x03b55a to 0x003aec (0x37a6d bytes) 000592 1786464732.821602 trunc from 0x003aed to 0x02f0bd (0x2b5d1 bytes) 000593 1786464732.835934 mapread 0x029280 thru 0x02e6a5 (0x5426 bytes) 000595 1786464732.847067 read 0x013cd5 thru 0x01c6fd (0x8a29 bytes) 000597 1786464732.848578 trunc from 0x02f0be to 0x0145b1 (0x1ab0c bytes) 000598 1786464732.863180 write 0x003fa1 thru 0x00f34d (0xb3ad bytes) 000599 1786464732.864914 write 0x01a2fd thru 0x024582 (0xa286 bytes) 000603 1786464732.869169 read 0x00b825 thru 0x0126c1 (0x6e9d bytes) 000605 1786464732.870267 mapwrite 0x005b52 thru 0x011864 (0xbd13 bytes) 000606 1786464732.913254 mapwrite 0x00a07d thru 0x00a9f5 (0x979 bytes) 000607 1786464732.947980 read 0x008edb thru 0x011e98 (0x8fbe bytes) 000609 1786464732.954303 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000610 1786464733.070576 mapread 0x01ec4b thru 0x024582 (0x5938 bytes) 000614 1786464733.093103 trunc from 0x024583 to 0x00bb9a (0x189e8 bytes) 000615 1786464733.115668 read 0x007c59 thru 0x00bb9a (0x3f42 bytes) 000616 1786464733.125692 write 0x02ff7a thru 0x0392fe (0x9385 bytes) 000617 1786464733.127778 write 0x027c9b thru 0x030e4c (0x91b2 bytes) 000618 1786464733.135375 read 0x00bfc0 thru 0x016999 (0xa9da bytes) 000622 1786464733.144845 mapwrite 0x00461f thru 0x013192 (0xeb74 bytes) 000624 1786464733.185929 mapwrite 0x0153fe thru 0x01f7b9 (0xa3bc bytes) 000625 1786464733.227613 mapwrite 0x01ce9f thru 0x01f9fc (0x2b5e bytes) 000627 1786464733.264486 mapwrite 0x01269e thru 0x01711b (0x4a7e bytes) 000629 1786464733.315562 write 0x035df2 thru 0x03ffff (0xa20e bytes) 000630 1786464733.316906 mapread 0x00222c thru 0x00dd9d (0xbb72 bytes) 000631 1786464733.322748 mapread 0x01be64 thru 0x021043 (0x51e0 bytes) 000632 1786464733.325677 trunc from 0x040000 to 0x007ed7 (0x38128 bytes) 000634 1786464733.340740 mapwrite 0x006298 thru 0x014e9a (0xec03 bytes) 000636 1786464733.477452 mapwrite 0x0215ea thru 0x02ed0a (0xd721 bytes) 000637 1786464733.604992 trunc from 0x02ed0b to 0x005cb2 (0x29058 bytes) 000639 1786464733.621914 mapread 0x003f93 thru 0x005cb2 (0x1d20 bytes) 000645 1786464733.629936 mapread 0x002c76 thru 0x005cb2 (0x303d bytes) 000646 1786464733.631965 read 0x002963 thru 0x005cb2 (0x3350 bytes) 000649 1786464733.634451 read 0x004a59 thru 0x005cb2 (0x125a bytes) 000651 1786464733.636101 read 0x00072a thru 0x005cb2 (0x5589 bytes) 000652 1786464733.636923 read 0x000d42 thru 0x005cb2 (0x4f71 bytes) 000653 1786464733.637744 mapwrite 0x01141f thru 0x01dc64 (0xc846 bytes) 000655 1786464733.720451 MIRROR_EXTEND to 2 mirrors 000657 1786464734.292925 trunc from 0x01dc65 to 0x02cfae (0xf34a bytes) 000659 1786464734.368487 write 0x011eda thru 0x0205cf (0xe6f6 bytes) 000661 1786464734.379353 write 0x00c469 thru 0x01724c (0xade4 bytes) 000662 1786464734.388145 mapwrite 0x00a87c thru 0x00ccbd (0x2442 bytes) 000664 1786464734.449937 mapread 0x0268c3 thru 0x02cfae (0x66ec bytes) 000666 1786464734.459864 write 0x0388a2 thru 0x03ffff (0x775e bytes) 000668 1786464734.461947 mapwrite 0x034163 thru 0x03ffff (0xbe9d bytes) 000670 1786464734.512984 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000674 1786464734.620775 read 0x034945 thru 0x03ffff (0xb6bb bytes) 000676 1786464734.628734 read 0x02119f thru 0x025bbd (0x4a1f bytes) 000677 1786464734.642395 MIRROR_EXTEND to 2 mirrors 000678 1786464735.097367 mapwrite 0x01289f thru 0x01fb72 (0xd2d4 bytes) 000680 1786464735.236952 mapread 0x019d4e thru 0x01f395 (0x5648 bytes) 000683 1786464735.241537 write 0x03f2af thru 0x03ffff (0xd51 bytes) 000685 1786464735.247943 write 0x033138 thru 0x03ffff (0xcec8 bytes) 000687 1786464735.253654 read 0x0392ae thru 0x03ffff (0x6d52 bytes) 000690 1786464735.256989 read 0x01347e thru 0x014ffc (0x1b7f bytes) 000693 1786464735.260817 trunc from 0x040000 to 0x022860 (0x1d79f bytes) 000694 1786464735.308667 mapwrite 0x025c29 thru 0x02a7ca (0x4ba2 bytes) 000695 1786464735.384061 read 0x01ee15 thru 0x020ddb (0x1fc7 bytes) 000697 1786464735.385320 write 0x002005 thru 0x004f23 (0x2f1f bytes) 000698 1786464735.388805 mapwrite 0x034f6e thru 0x035ca9 (0xd3c bytes) 000700 1786464735.440149 read 0x01b842 thru 0x01c542 (0xd01 bytes) 000701 1786464735.440838 mapread 0x02fe02 thru 0x035ca9 (0x5ea8 bytes) 000704 1786464735.443823 mapread 0x017776 thru 0x019912 (0x219d bytes) 000705 1786464735.444895 read 0x01ac2c thru 0x01e865 (0x3c3a bytes) 000709 1786464735.447567 write 0x01a299 thru 0x029582 (0xf2ea bytes) 000711 1786464735.449740 read 0x00cbee thru 0x00ed1c (0x212f bytes) 000712 1786464735.450461 write 0x034931 thru 0x03bf71 (0x7641 bytes) 000715 1786464735.456709 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bf72): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf72): Mirror 2: 0x71086b98 (stale) CRC-32 checksum value for chunk [0, 0x3bf72): Mirror 1: 0xa7188cc3 Mirror 3: 0xa7188cc3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf72: skipped 000716 1786464736.076325 mapwrite 0x039d8b thru 0x03ffff (0x6275 bytes) 000717 1786464736.249276 mapread 0x034d5d thru 0x039929 (0x4bcd bytes) 000718 1786464736.250889 mapwrite 0x030580 thru 0x03c246 (0xbcc7 bytes) 000719 1786464736.308490 mapwrite 0x01a19a thru 0x01cc03 (0x2a6a bytes) 000720 1786464736.354615 trunc from 0x040000 to 0x019bf3 (0x2640c bytes) 000724 1786464736.373437 write 0x03f741 thru 0x03ffff (0x8bf bytes) 000727 1786464736.376773 read 0x026cf7 thru 0x03614b (0xf455 bytes) 000728 1786464736.386180 mapread 0x00bad4 thru 0x00da28 (0x1f55 bytes) 000729 1786464736.393768 mapread 0x035376 thru 0x03fa70 (0xa6fb bytes) 000730 1786464736.404247 mapwrite 0x0000be thru 0x005a64 (0x59a7 bytes) 000732 1786464736.443548 mapread 0x01f385 thru 0x02e093 (0xed0f bytes) 000734 1786464736.451202 mapread 0x015f53 thru 0x01bbc8 (0x5c76 bytes) 000735 1786464736.456230 MIRROR_RESYNC 3 mirrors 000736 1786464737.115194 trunc from 0x040000 to 0x03c1c1 (0x3e3e bytes) 000737 1786464737.189927 mapread 0x01269d thru 0x0143d6 (0x1d3a bytes) 000739 1786464737.206160 write 0x034f9f thru 0x039280 (0x42e2 bytes) 000740 1786464737.213858 trunc from 0x03c1c2 to 0x014bb4 (0x2760d bytes) 000741 1786464737.230695 write 0x0172be thru 0x025849 (0xe58c bytes) 000743 1786464737.233568 trunc from 0x02584a to 0x0342ae (0xea65 bytes) 000745 1786464737.251858 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x342af): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342af): Mirror 2: 0xe292e1d (stale) Mirror 3: 0xe292e1d (stale) CRC-32 checksum value for chunk [0, 0x342af): Mirror 1: 0x7ee6d897 Mirror 4: 0x7ee6d897 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342af: skipped 000746 1786464737.938780 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x342af): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342af): Mirror 2: 0xe292e1d (stale) Mirror 3: 0xe292e1d (stale) CRC-32 checksum value for chunk [0, 0x342af): Mirror 1: 0x7ee6d897 Mirror 4: 0x7ee6d897 Mirror 5: 0x7ee6d897 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342af: skipped 000747 1786464738.595862 read 0x02ddab thru 0x0342ae (0x6504 bytes) 000749 1786464738.602418 trunc from 0x0342af to 0x022b1f (0x1178f bytes) 000750 1786464738.670861 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x22b20): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22b20): Mirror 2: 0x488c6732 (stale) Mirror 3: 0x488c6732 (stale) Mirror 4: 0x4a65e7da (stale) Mirror 5: 0x4a65e7da (stale) CRC-32 checksum value for chunk [0, 0x22b20): Mirror 1: 0x4a65e7da Mirror 6: 0x4a65e7da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22b20: skipped 000752 1786464739.316466 MIRROR_RESYNC 6 mirrors 000754 1786464739.996194 mapwrite 0x0108a1 thru 0x019df3 (0x9553 bytes) 000755 1786464740.144421 trunc from 0x022b20 to 0x0269af (0x3e90 bytes) 000756 1786464740.163278 trunc from 0x0269b0 to 0x0039f6 (0x22fb9 bytes) 000757 1786464740.178984 MIRROR_RESYNC 6 mirrors 000763 1786464740.851306 MIRROR_RESYNC 6 mirrors 000765 1786464741.093576 trunc from 0x0039f7 to 0x02f611 (0x2bc1b bytes) 000767 1786464741.146218 write 0x03b11b thru 0x03ffff (0x4ee5 bytes) 000768 1786464741.147800 write 0x0226e5 thru 0x029e91 (0x77ad bytes) 000770 1786464741.159576 write 0x034efc thru 0x037201 (0x2306 bytes) 000771 1786464741.168929 mapread 0x026de1 thru 0x02b506 (0x4726 bytes) 000772 1786464741.180966 read 0x001f9d thru 0x00f5a1 (0xd605 bytes) 000774 1786464741.192123 mapwrite 0x038db9 thru 0x03ffff (0x7247 bytes) 000775 1786464741.248980 mapwrite 0x01943b thru 0x01b363 (0x1f29 bytes) 000776 1786464741.291259 write 0x0304dc thru 0x03ffd0 (0xfaf5 bytes) 000777 1786464741.297120 mapread 0x025e6d thru 0x02721c (0x13b0 bytes) 000778 1786464741.300477 write 0x010214 thru 0x013ee2 (0x3ccf bytes) 000779 1786464741.305816 read 0x03996e thru 0x03ffff (0x6692 bytes) 000780 1786464741.310438 MIRROR_RESYNC 6 mirrors 000781 1786464743.124419 read 0x004ee6 thru 0x01004a (0xb165 bytes) 000782 1786464743.137404 MIRROR_RESYNC 6 mirrors 000784 1786464743.422631 mapread 0x02c3d5 thru 0x039711 (0xd33d bytes) 000785 1786464743.442547 mapread 0x02cfd4 thru 0x039664 (0xc691 bytes) 000787 1786464743.443570 read 0x039ad9 thru 0x03ffff (0x6527 bytes) 000788 1786464743.443863 mapwrite 0x0164e0 thru 0x01ab4d (0x466e bytes) 000790 1786464743.551057 mapread 0x027519 thru 0x0306cc (0x91b4 bytes) 000791 1786464743.564634 mapread 0x01e8a0 thru 0x023664 (0x4dc5 bytes) 000794 1786464743.575556 write 0x03a2b2 thru 0x03a655 (0x3a4 bytes) 000796 1786464743.583623 mapread 0x01f155 thru 0x027489 (0x8335 bytes) 000798 1786464743.586339 write 0x023782 thru 0x02f84e (0xc0cd bytes) 000800 1786464743.590799 trunc from 0x040000 to 0x0103a4 (0x2fc5b bytes) 000801 1786464743.613192 write 0x0377fd thru 0x03dbed (0x63f1 bytes) 000802 1786464743.614996 trunc from 0x03dbee to 0x027f34 (0x15cb9 bytes) 000804 1786464743.630588 write 0x025761 thru 0x0329f6 (0xd296 bytes) 000805 1786464743.638658 trunc from 0x0329f7 to 0x02e811 (0x41e5 bytes) 000809 1786464743.661660 mapread 0x01640e thru 0x024b82 (0xe775 bytes) 000810 1786464743.679348 trunc from 0x02e812 to 0x00279f (0x2c072 bytes) 000812 1786464743.698073 mapwrite 0x01ac61 thru 0x01b62e (0x9ce bytes) 000813 1786464743.751597 write 0x00f50e thru 0x01e190 (0xec83 bytes) 000815 1786464743.762320 mapwrite 0x006638 thru 0x00fcfb (0x96c4 bytes) 000816 1786464743.812346 mapread 0x01537f thru 0x017bc6 (0x2848 bytes) 000820 1786464743.816869 mapwrite 0x00c8d8 thru 0x012810 (0x5f39 bytes) 000821 1786464743.864490 write 0x023fa1 thru 0x0329fc (0xea5c bytes) 000822 1786464743.870055 mapwrite 0x00c897 thru 0x01c3e2 (0xfb4c bytes) 000823 1786464743.923650 read 0x0028f7 thru 0x004593 (0x1c9d bytes) 000824 1786464743.924610 mapwrite 0x000a54 thru 0x0024c2 (0x1a6f bytes) 000827 1786464743.965437 write 0x025947 thru 0x028161 (0x281b bytes) 000828 1786464743.967301 trunc from 0x0329fd to 0x00071e (0x322de bytes) 000829 1786464743.995377 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x71f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x71f): Mirror 2: 0x6985a904 (stale) Mirror 3: 0x6985a904 (stale) Mirror 4: 0x6985a904 (stale) Mirror 5: 0x6985a904 (stale) Mirror 6: 0x6985a904 (stale) CRC-32 checksum value for chunk [0, 0x71f): Mirror 1: 0x6985a904 Mirror 7: 0x6985a904 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x71f: skipped 000830 1786464744.592860 MIRROR_RESYNC 7 mirrors 000832 1786464745.265349 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000833 1786464745.529230 MIRROR_RESYNC 6 mirrors 000834 1786464745.775687 mapread 0x0003d8 thru 0x00071e (0x347 bytes) 000835 1786464745.785609 read 0x0000cc thru 0x00071e (0x653 bytes) 000836 1786464745.792315 MIRROR_EXTEND to 7 mirrors 000837 1786464746.260127 trunc from 0x00071f to 0x01467e (0x13f60 bytes) 000838 1786464746.330889 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1467f): Mirror 3: 0x2ba7216c (stale) Mirror 4: 0x2ba7216c (stale) Mirror 5: 0x2ba7216c (stale) Mirror 6: 0x2ba7216c (stale) Mirror 8: 0x2ba7216c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1467f): Mirror 3: 0x2ba7216c (stale) Mirror 4: 0x2ba7216c (stale) Mirror 5: 0x2ba7216c (stale) Mirror 6: 0x2ba7216c (stale) Mirror 8: 0x2ba7216c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1467f: skipped 000839 1786464746.614269 mapwrite 0x01b064 thru 0x01f33e (0x42db bytes) 000841 1786464746.690871 write 0x02ac39 thru 0x0309dc (0x5da4 bytes) 000843 1786464746.694710 mapwrite 0x007b0c thru 0x00f78c (0x7c81 bytes) 000844 1786464746.743184 trunc from 0x0309dd to 0x030fc4 (0x5e8 bytes) 000849 1786464746.759489 mapread 0x020ae9 thru 0x023b65 (0x307d bytes) 000851 1786464746.768289 write 0x02979a thru 0x0311da (0x7a41 bytes) 000853 1786464746.775689 mapwrite 0x02e2f3 thru 0x033f58 (0x5c66 bytes) 000854 1786464746.821833 mapwrite 0x003c37 thru 0x003cae (0x78 bytes) 000856 1786464746.851560 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33f59): Mirror 3: 0xc61e2169 (stale) Mirror 4: 0xc61e2169 (stale) Mirror 5: 0xc61e2169 (stale) Mirror 6: 0xc61e2169 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f59): Mirror 3: 0xc61e2169 (stale) Mirror 4: 0xc61e2169 (stale) Mirror 5: 0xc61e2169 (stale) Mirror 6: 0xc61e2169 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f59: skipped 000857 1786464747.127889 mapread 0x024474 thru 0x02822f (0x3dbc bytes) 000862 1786464747.151186 MIRROR_RESYNC 5 mirrors 000863 1786464748.179383 read 0x023c48 thru 0x02f365 (0xb71e bytes) 000865 1786464748.186630 mapwrite 0x029df2 thru 0x02c121 (0x2330 bytes) 000867 1786464748.303535 trunc from 0x033f59 to 0x005171 (0x2ede7 bytes) 000869 1786464748.317623 mapread 0x002476 thru 0x005171 (0x2cfc bytes) 000871 1786464748.327790 trunc from 0x005172 to 0x0396e6 (0x34575 bytes) 000872 1786464748.346790 mapread 0x021e31 thru 0x02d7e3 (0xb9b3 bytes) 000873 1786464748.372856 mapwrite 0x02283a thru 0x024a25 (0x21ec bytes) 000874 1786464748.413427 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x396e7): Mirror 4: 0x3e1b272e (stale) Mirror 5: 0x3e1b272e (stale) Mirror 6: 0x3e1b272e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396e7): Mirror 4: 0x3e1b272e (stale) Mirror 5: 0x3e1b272e (stale) Mirror 6: 0x3e1b272e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396e7: skipped 000876 1786464748.659941 trunc from 0x0396e7 to 0x004925 (0x34dc1 bytes) 000880 1786464748.671511 mapread 0x002ec8 thru 0x004925 (0x1a5e bytes) 000881 1786464748.684727 mapread 0x002eea thru 0x004925 (0x1a3c bytes) 000885 1786464748.687542 write 0x02abf1 thru 0x02ea97 (0x3ea7 bytes) 000888 1786464748.689802 mapwrite 0x02202f thru 0x027097 (0x5069 bytes) 000889 1786464748.739813 trunc from 0x02ea98 to 0x01505f (0x19a38 bytes) 000890 1786464748.755692 mapread 0x014806 thru 0x01505f (0x85a bytes) 000891 1786464748.765267 mapwrite 0x00a1f8 thru 0x00b710 (0x1519 bytes) 000892 1786464748.803787 mapread 0x003fe9 thru 0x0124b5 (0xe4cd bytes) 000893 1786464748.805235 read 0x00413e thru 0x011636 (0xd4f9 bytes) 000896 1786464748.807467 read 0x005db8 thru 0x00807d (0x22c6 bytes) 000897 1786464748.808159 trunc from 0x015060 to 0x00b30e (0x9d51 bytes) 000898 1786464748.825075 write 0x02156f thru 0x0221cf (0xc61 bytes) 000903 1786464748.829285 read 0x0032c4 thru 0x00dbe7 (0xa924 bytes) 000904 1786464748.835321 read 0x0124c2 thru 0x01b934 (0x9473 bytes) 000905 1786464748.838712 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x221d0): Mirror 5: 0x4ca1ae48 (stale) Mirror 6: 0x4ca1ae48 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x221d0): Mirror 5: 0x4ca1ae48 (stale) Mirror 6: 0x4ca1ae48 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x221d0: skipped 000907 1786464749.049387 mapwrite 0x009b13 thru 0x0154f8 (0xb9e6 bytes) 000908 1786464749.103635 read 0x00b61a thru 0x0111b6 (0x5b9d bytes) 000909 1786464749.108083 MIRROR_RESYNC 3 mirrors 000911 1786464749.721747 trunc from 0x0221d0 to 0x035986 (0x137b7 bytes) 000913 1786464749.799442 trunc from 0x035987 to 0x012b66 (0x22e20 bytes) 000914 1786464749.815962 MIRROR_RESYNC 3 mirrors 000916 1786464750.302962 MIRROR_EXTEND to 4 mirrors 000917 1786464750.732457 mapwrite 0x01fe27 thru 0x021a66 (0x1c40 bytes) 000918 1786464750.846264 mapread 0x01dd74 thru 0x0205a7 (0x2834 bytes) 000919 1786464750.847348 mapwrite 0x028b57 thru 0x035352 (0xc7fc bytes) 000921 1786464750.929240 MIRROR_RESYNC 4 mirrors 000922 1786464751.858377 mapread 0x00180a thru 0x001be1 (0x3d8 bytes) 000925 1786464751.863689 mapread 0x02fb4c thru 0x035352 (0x5807 bytes) 000926 1786464751.875320 mapwrite 0x00daca thru 0x019b98 (0xc0cf bytes) 000928 1786464752.017946 read 0x018cfc thru 0x01ebcb (0x5ed0 bytes) 000929 1786464752.018680 read 0x0288e9 thru 0x02b7d0 (0x2ee8 bytes) 000930 1786464752.030716 trunc from 0x035353 to 0x026bba (0xe798 bytes) 000932 1786464752.048798 write 0x024ce1 thru 0x02b378 (0x6698 bytes) 000933 1786464752.050425 read 0x01fc26 thru 0x02b109 (0xb4e4 bytes) 000934 1786464752.051462 read 0x00322b thru 0x0040a2 (0xe78 bytes) 000935 1786464752.052181 read 0x0049e1 thru 0x00cef8 (0x8518 bytes) 000936 1786464752.053254 trunc from 0x02b379 to 0x00ebad (0x1c7cb bytes) 000937 1786464752.070919 write 0x01c939 thru 0x02adb0 (0xe478 bytes) 000939 1786464752.073813 trunc from 0x02adb1 to 0x01a155 (0x10c5b bytes) 000943 1786464752.101133 mapwrite 0x033805 thru 0x03c9ae (0x91aa bytes) 000946 1786464752.159713 mapread 0x034a8c thru 0x035929 (0xe9e bytes) 000947 1786464752.165387 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c9af): Mirror 6: 0xf028aa77 (stale) Mirror 7: 0xf028aa77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9af): Mirror 6: 0xf028aa77 (stale) Mirror 7: 0xf028aa77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9af: skipped 000948 1786464752.403846 MIRROR_RESYNC 3 mirrors 000950 1786464752.993444 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x227:0x0]: Invalid argument (22) 000951 1786464753.194521 mapwrite 0x03f681 thru 0x03ffff (0x97f bytes) 000958 1786464753.323400 write 0x0195bb thru 0x01eedb (0x5921 bytes) 000961 1786464753.333305 trunc from 0x040000 to 0x0068cd (0x39732 bytes) 000962 1786464753.348958 mapread 0x00019b thru 0x0068cd (0x6733 bytes) 000964 1786464753.360995 mapread 0x0019e7 thru 0x0068cd (0x4ee7 bytes) 000968 1786464753.364642 mapread 0x004163 thru 0x0061e0 (0x207e bytes) 000969 1786464753.365632 trunc from 0x0068ce to 0x02e674 (0x27da7 bytes) 000970 1786464753.377730 mapwrite 0x00c1b2 thru 0x0196de (0xd52d bytes) 000971 1786464753.439989 MIRROR_RESYNC 2 mirrors 000972 1786464754.003427 mapwrite 0x0181d2 thru 0x01bfd4 (0x3e03 bytes) 000975 1786464754.115333 write 0x0265be thru 0x035148 (0xeb8b bytes) 000978 1786464754.124930 write 0x00abd6 thru 0x01a709 (0xfb34 bytes) 000979 1786464754.131927 mapread 0x0230e7 thru 0x02f137 (0xc051 bytes) 000980 1786464754.139687 read 0x00b65c thru 0x016ecc (0xb871 bytes) 000981 1786464754.140983 mapwrite 0x012d15 thru 0x01c9d0 (0x9cbc bytes) 000984 1786464754.183491 read 0x032280 thru 0x035148 (0x2ec9 bytes) 000985 1786464754.184623 write 0x03cbb4 thru 0x03d59b (0x9e8 bytes) 000986 1786464754.190234 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d59c): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d59c): Mirror 6: 0x12e7f2f3 (stale) CRC-32 checksum value for chunk [0, 0x3d59c): Mirror 1: 0x6baf6fb2 Mirror 7: 0x6baf6fb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d59c: skipped 000988 1786464754.881134 mapread 0x00bb9f thru 0x019d27 (0xe189 bytes) 000991 1786464754.955844 MIRROR_RESYNC 3 mirrors 000992 1786464755.650784 MIRROR_RESYNC 3 mirrors 000996 1786464755.929770 read 0x01e274 thru 0x02d4c2 (0xf24f bytes) 000997 1786464755.942249 trunc from 0x03d59c to 0x02843c (0x1515f bytes) 000998 1786464756.041301 mapread 0x011851 thru 0x0146d1 (0x2e81 bytes) 000999 1786464756.067303 MIRROR_RESYNC 3 mirrors All operations completed A-OK! PASS 70a (58s) == sanity-flr test 71: check mirror extend parent fid ==== 12:12:39 (1786464759) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 12:12:41 (1786464761) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137079 s, 15.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105027 s, 10.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x37f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x380:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (5s) == sanity-flr test 200a: stress test ===================== 12:12:46 (1786464766) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2830240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 5981344 .. Extending file size to 7696064 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6405088 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8382752 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7150656 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8955520 .. Extending file size to 8606784 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8642336 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8673376 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8694656 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57599 57601 57602 57603 57604 done 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) 192.168.204.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (72s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:13:58 (1786464838) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1337472 .. Extending file size to 6243808 .. done Extending file size to 4870496 .. Extending file size to 8408480 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7989792 .. Extending file size to 9116128 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8581600 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8613952 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9008672 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9238496 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 59587 59589 59590 59593 59594 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) 192.168.204.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (70s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:15:08 (1786464908) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0501231 s, 41.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135197 s, 15.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x14f568b8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:15:16 (1786464916) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.037838 s, 55.4 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 12:15:19 (1786464919) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b2:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x395:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:15:22 (1786464922) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0871993 s, 24.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x396:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x397:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:15:25 (1786464925) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0854673 s, 24.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x398:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x398:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x399:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:15:28 (1786464928) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.072003 s, 29.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39b:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:15:53 (1786464953) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395251 s, 26.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b8:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:16:38 (1786464998) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ba:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.066274 s, 31.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3b9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ba:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:17:24 (1786465044) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. Creating new pool test_204f oleg405-server: Pool lustre.test_204f created Adding targets to pool oleg405-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg405-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0522314 s, 40.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg405-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg405-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg405-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (59s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:18:23 (1786465103) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:18:26 (1786465106) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:18:29 (1786465109) oleg405-server: Pool lustre.test_206 created Adding targets to pool oleg405-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg405-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg405-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg405-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg405-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:18:44 (1786465124) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306368 s, 34.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122632 s, 85.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf36868c2 Mirror 3: 0xf36868c2 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:18:48 (1786465128) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:18:49 (1786465129) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:18:51 (1786465131) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0400076 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x23e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x11e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x11e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x11e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x22e0000 Read this (4 bytes): BBBB 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:19:08 (1786465148) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00901223 s, 909 kB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:19:10 (1786465150) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178 s, 58.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x28f lmm_fid: [0x200000404:0x28f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x28f lmm_fid: [0x200000404:0x28f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3aa:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834575 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-11 12:19:10.000000000 -0400 Modify: 2026-08-11 12:19:10.000000000 -0400 Change: 2026-08-11 12:19:11.000000000 -0400 Birth: 2026-08-11 12:19:11.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:19:13 (1786465153) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:19:14 (1786465154) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.128785 s, 318 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0172483 s, 237 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 11 12:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x291:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 11 12:19 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:19:16 (1786465156) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:19:19 (1786465159) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 939 sec ============ 12:19:22 (1786465162) === sanity-flr: start cleanup 12:19:23 (1786465163) === === sanity-flr: finish cleanup 12:19:25 (1786465165) ===