-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 11:12:40 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg228-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 11:12:56 (1788621176) === oleg228-client.virtnet: executing check_config_client /mnt/lustre oleg228-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg228-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88c586c63800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88c586c63800.idle_timeout=debug disable quota as required oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 11:13:27 (1788621207) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7631] == sanity-flr test 0a: lfs mirror create with -N option == 11:13:29 (1788621209) PASS 0a (12s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 11:13:41 (1788621221) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 11:13:43 (1788621223) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 11:13:46 (1788621226) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 11:14:05 (1788621245) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 11:14:08 (1788621248) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 11:14:10 (1788621250) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88c586c63800.stats=clear osc.lustre-OST0001-osc-ffff88c586c63800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 11:14:17 (1788621257) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 11:14:27 (1788621267) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 11:14:36 (1788621276) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 11:14:39 (1788621279) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 11:14:48 (1788621288) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 11:14:59 (1788621299) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 11:15:06 (1788621306) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 11:15:11 (1788621311) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 11:15:19 (1788621319) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 11:15:28 (1788621328) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 11:15:36 (1788621336) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 1.58123 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698172 s, 15.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 11:15:46 (1788621346) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.131992 s, 23.8 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 11:15:53 (1788621353) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 11:16:03 (1788621363) Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88c586c63800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88c586c63800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88c586c63800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88c586c63800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:17:14 (1788621434) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.499521 s, 21.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88c586c63800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88c586c63800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:17:59 (1788621479) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88c586c63800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88c586c63800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88c586c63800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88c586c63800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:19:17 (1788621557) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.7655 s, 13.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 11:19 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.920s user 0m0.002s sys 0m0.149s Stopping /mnt/lustre-ost1 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88c586c63800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88c586c63800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.279s user 0m0.001s sys 0m0.177s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg228-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg228-server oleg228-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88c586c63800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88c586c63800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.768s user 0m0.002s sys 0m0.186s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg228-server: oleg228-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg228-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88c586c63800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88c586c63800.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:20:58 (1788621658) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:21:02 (1788621662) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:21:05 (1788621665) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 11:21:08 (1788621668) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 11:21:15 (1788621675) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.394905 s, 31.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:21:26 (1788621686) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07272 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401504 s, 26.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:21:34 (1788621694) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:21:42 (1788621702) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000889132 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000721813 s, 1.4 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000620304 s, 1.7 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00143659 s, 713 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 11:21:52 (1788621712) create test files with size 15731584 .. 64b5a564838048ff0e23035267120086 - e7e8e3ba771d8e18ef55b3eeabfa67e6 - e15fd5e52ae7f4e1baf0412ce5de0cff - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 11:22:12 (1788621732) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:22:27 (1788621747) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:22:33 (1788621753) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975365 s, 10.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261577 s, 40.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 11:22:41 (1788621761) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194044 s, 10.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.318571 s, 13.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2204 s, 9.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 11:22:58 (1788621778) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 11:23:02 (1788621782) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:23:05 (1788621785) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497113 s, 21.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 11:23:13 (1788621793) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:23:17 (1788621797) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xc7:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:23:27 (1788621807) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.36168 s, 29.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:23:40 (1788621820) llite.lustre-ffff88c5832f5800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.346249 s, 30.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg228-client.virtnet /mnt/lustre (opts:) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff88c58508b800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 11:23:55 (1788621835) error lfs split: create victim file failed: File exists PASS 44f (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:24:10 (1788621850) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:24:54 (1788621894) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:25:37 (1788621937) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 11:25:41 (1788621941) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.252945 s, 12.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.2367 s, 13.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1a803386 (nosync,stale) Mirror 2: 0x1a803386 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6229f145af4b38c16875204022805955 - mirror 1 checksum: d12949fb8b0c57702a90684f2f0b729a - mirror 2 checksum: 6229f145af4b38c16875204022805955 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T11:25:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x125:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T11:25:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x126:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6229f145af4b38c16875204022805955 - mirror 1 checksum: d12949fb8b0c57702a90684f2f0b729a - mirror 2 checksum: d12949fb8b0c57702a90684f2f0b729a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T11:25:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x125:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T11:25:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x126:0x0] } PASS 48 (16s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:25:59 (1788621959) Failing mds1 on oleg228-server Stopping /mnt/lustre-mds1 (opts:) on oleg228-server 11:26:07 (1788621967) shut down facet: mds1 facet_host: oleg228-server facet_failover_host: oleg228-server Failover mds1 to oleg228-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 11:26:29 (1788621989) targets are mounted 11:26:29 (1788621989) facet_failover done oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155206 s, 67.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x69 lmm_fid: [0x200000404:0x69:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x127:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:26:44 (1788622004) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:26:48 (1788622008) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:26:50 (1788622010) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00859009 s, 1.9 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:26:59 (1788622019) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:27:03 (1788622023) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0111566 s, 367 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0593167 s, 69.1 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:27:10 (1788622030) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:27:30 (1788622050) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:27:58 (1788622078) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa7:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:28:22 (1788622102) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:28:45 (1788622125) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 11:28:52 (1788622132) mirror create pid 54979 mirror split pid 54980 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54979 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54980 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 11:29:59 (1788622199) Seed set to 2201 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1788622201.669011 write 0x02ad97 thru 0x02d621 (0x288b bytes) 000002 1788622201.675334 read 0x027736 thru 0x02bf4c (0x4817 bytes) 000003 1788622201.679502 write 0x02e247 thru 0x02e675 (0x42f bytes) 000004 1788622201.684502 mapwrite 0x00a1ca thru 0x00aa56 (0x88d bytes) 000005 1788622201.740858 mapread 0x007590 thru 0x016a4d (0xf4be bytes) 000006 1788622201.764299 read 0x015d3a thru 0x021ef3 (0xc1ba bytes) truncating to largest ever: 0x1b716 000010 1788622201.781601 trunc from 0x02e676 to 0x01b715 (0x12f60 bytes) 000011 1788622201.801735 write 0x014bbe thru 0x01698e (0x1dd1 bytes) 000013 1788622201.804911 mapread 0x0024ff thru 0x00bdcc (0x98ce bytes) 000015 1788622201.811338 MIRROR_EXTEND to 2 mirrors 000019 1788622203.021403 mapwrite 0x01a7e9 thru 0x02617e (0xb996 bytes) 000021 1788622203.201475 MIRROR_RESYNC 2 mirrors 000022 1788622203.951399 write 0x01d0f3 thru 0x025d54 (0x8c62 bytes) 000023 1788622204.084352 read 0x003933 thru 0x011107 (0xd7d5 bytes) 000024 1788622204.110692 mapread 0x008012 thru 0x0085fa (0x5e9 bytes) 000026 1788622204.120600 read 0x0161f9 thru 0x01f7a2 (0x95aa bytes) 000027 1788622204.134288 mapread 0x00eb04 thru 0x0176da (0x8bd7 bytes) 000028 1788622204.135508 read 0x00750a thru 0x014fa7 (0xda9e bytes) 000029 1788622204.137328 mapread 0x019d87 thru 0x01cc66 (0x2ee0 bytes) 000030 1788622204.138292 write 0x0284af thru 0x034f36 (0xca88 bytes) 000032 1788622204.146539 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000035 1788622204.320040 write 0x01bdf7 thru 0x01e76a (0x2974 bytes) 000037 1788622204.335959 read 0x02c3aa thru 0x02ecfc (0x2953 bytes) 000042 1788622204.346368 mapwrite 0x010b6e thru 0x016cad (0x6140 bytes) 000044 1788622204.407452 mapread 0x01f963 thru 0x020134 (0x7d2 bytes) 000045 1788622204.415987 mapread 0x023220 thru 0x028db0 (0x5b91 bytes) 000046 1788622204.423596 mapread 0x01f2c6 thru 0x02ef1e (0xfc59 bytes) 000048 1788622204.425440 write 0x01d535 thru 0x026e81 (0x994d bytes) 000049 1788622204.430003 read 0x0001bd thru 0x001651 (0x1495 bytes) 000051 1788622204.438401 read 0x012bdb thru 0x01f757 (0xcb7d bytes) 000052 1788622204.439237 mapwrite 0x0040e0 thru 0x00e8fa (0xa81b bytes) 000055 1788622204.481765 mapwrite 0x02dd72 thru 0x0383e0 (0xa66f bytes) 000056 1788622204.575826 read 0x02f285 thru 0x0383e0 (0x915c bytes) 000057 1788622204.577327 read 0x008b31 thru 0x015ffe (0xd4ce bytes) 000058 1788622204.578917 read 0x02adb1 thru 0x0383e0 (0xd630 bytes) 000059 1788622204.580043 read 0x00fe56 thru 0x011487 (0x1632 bytes) 000065 1788622204.586354 write 0x005a4d thru 0x008b3b (0x30ef bytes) 000068 1788622204.590325 read 0x00da5d thru 0x015ea9 (0x844d bytes) 000069 1788622204.593568 mapwrite 0x020b93 thru 0x020e29 (0x297 bytes) 000071 1788622204.650517 mapwrite 0x012da1 thru 0x01f539 (0xc799 bytes) 000072 1788622204.713057 mapread 0x00ab40 thru 0x0192aa (0xe76b bytes) 000073 1788622204.716053 mapwrite 0x000af5 thru 0x00ff61 (0xf46d bytes) 000074 1788622204.782508 mapread 0x0153fc thru 0x021fb3 (0xcbb8 bytes) 000075 1788622204.783424 trunc from 0x0383e1 to 0x00612e (0x322b2 bytes) 000076 1788622204.803700 mapread 0x000185 thru 0x00612e (0x5faa bytes) 000077 1788622204.805098 mapwrite 0x002f8d thru 0x011af7 (0xeb6b bytes) 000078 1788622204.876641 mapread 0x002744 thru 0x009c05 (0x74c2 bytes) 000080 1788622204.878407 read 0x009a86 thru 0x011af7 (0x8072 bytes) 000083 1788622204.880804 trunc from 0x011af8 to 0x01449d (0x29a6 bytes) 000084 1788622204.893072 read 0x00c1bd thru 0x01449d (0x82e1 bytes) 000088 1788622204.901407 trunc from 0x01449e to 0x00821d (0xc280 bytes) 000089 1788622204.920433 write 0x02e758 thru 0x03645e (0x7d07 bytes) 000090 1788622204.925617 MIRROR_EXTEND to 2 mirrors 000091 1788622205.679428 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000092 1788622205.784952 mapwrite 0x01fafa thru 0x0299f3 (0x9efa bytes) 000094 1788622205.868225 read 0x017a58 thru 0x022713 (0xacbc bytes) 000095 1788622205.874245 read 0x013042 thru 0x0131dc (0x19b bytes) 000096 1788622205.884396 write 0x017d1e thru 0x0219b9 (0x9c9c bytes) 000097 1788622205.886061 read 0x00ef66 thru 0x00fbdf (0xc7a bytes) truncating to largest ever: 0x33f8e 000101 1788622205.889210 trunc from 0x03645f to 0x033f8d (0x24d1 bytes) truncating to largest ever: 0x3f517 000102 1788622205.911885 trunc from 0x033f8e to 0x03f516 (0xb589 bytes) 000103 1788622205.933243 trunc from 0x03f517 to 0x0135e3 (0x2bf33 bytes) 000104 1788622205.949948 mapread 0x010c0a thru 0x0135e3 (0x29da bytes) 000108 1788622205.952025 read 0x010614 thru 0x0135e3 (0x2fd0 bytes) 000109 1788622205.954482 mapread 0x006c88 thru 0x008f6e (0x22e7 bytes) 000110 1788622205.956374 trunc from 0x0135e4 to 0x02c4e2 (0x18eff bytes) 000111 1788622205.971544 mapwrite 0x03d798 thru 0x03ffff (0x2868 bytes) 000114 1788622206.039721 MIRROR_EXTEND to 2 mirrors 000116 1788622206.763932 write 0x016b93 thru 0x02048e (0x98fc bytes) 000117 1788622206.841268 trunc from 0x040000 to 0x038a12 (0x75ed bytes) 000119 1788622206.860405 mapwrite 0x0009dd thru 0x00cfd9 (0xc5fd bytes) 000120 1788622206.926499 read 0x01c2c5 thru 0x02a22b (0xdf67 bytes) 000121 1788622206.933588 trunc from 0x038a13 to 0x015316 (0x236fc bytes) 000123 1788622206.950434 write 0x03c105 thru 0x03ffff (0x3efb bytes) 000124 1788622206.951972 read 0x03a5ac thru 0x03ef02 (0x4957 bytes) 000126 1788622206.961415 mapread 0x000db7 thru 0x0053d8 (0x4622 bytes) 000128 1788622206.963229 mapread 0x03c4f1 thru 0x03dbba (0x16ca bytes) 000130 1788622206.967416 mapwrite 0x0001a2 thru 0x009520 (0x937f bytes) 000135 1788622207.078839 MIRROR_RESYNC 2 mirrors 000136 1788622208.105446 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000137 1788622208.212339 mapwrite 0x0301ae thru 0x0328ba (0x270d bytes) 000141 1788622208.278190 write 0x00da87 thru 0x00f783 (0x1cfd bytes) 000142 1788622208.283925 trunc from 0x040000 to 0x004348 (0x3bcb7 bytes) 000143 1788622208.297600 mapwrite 0x00ac63 thru 0x011aa7 (0x6e45 bytes) 000144 1788622208.357843 mapread 0x00f5fd thru 0x011aa7 (0x24ab bytes) 000146 1788622208.359153 mapwrite 0x003dae thru 0x0060e4 (0x2337 bytes) 000149 1788622208.400214 read 0x005a7d thru 0x00c4dd (0x6a61 bytes) 000150 1788622208.400877 trunc from 0x011aa8 to 0x030b40 (0x1f099 bytes) 000151 1788622208.421000 mapread 0x00f127 thru 0x01e05e (0xef38 bytes) 000157 1788622208.442843 write 0x011b21 thru 0x020b1f (0xefff bytes) 000161 1788622208.446543 read 0x004b08 thru 0x012e14 (0xe30d bytes) 000166 1788622208.451322 read 0x023159 thru 0x026821 (0x36c9 bytes) 000167 1788622208.460492 write 0x03f62f thru 0x03f6d3 (0xa5 bytes) 000169 1788622208.463201 trunc from 0x03f6d4 to 0x034930 (0xada3 bytes) 000171 1788622208.482373 read 0x004f96 thru 0x00b5d5 (0x6640 bytes) 000172 1788622208.483003 write 0x00b242 thru 0x00fdb6 (0x4b75 bytes) 000173 1788622208.484644 read 0x025705 thru 0x02de30 (0x872c bytes) 000175 1788622208.486860 mapwrite 0x02911d thru 0x0373ed (0xe2d1 bytes) 000177 1788622208.570735 trunc from 0x0373ee to 0x005f61 (0x3148c bytes) 000179 1788622208.587762 trunc from 0x005f62 to 0x026341 (0x203e0 bytes) 000181 1788622208.607300 mapwrite 0x00a69f thru 0x00e460 (0x3dc2 bytes) 000182 1788622208.683062 mapread 0x01c702 thru 0x022026 (0x5925 bytes) 000183 1788622208.730502 trunc from 0x026342 to 0x019ba6 (0xc79b bytes) 000184 1788622208.748797 read 0x016b4c thru 0x016bf1 (0xa6 bytes) 000185 1788622208.761028 MIRROR_EXTEND to 2 mirrors 000187 1788622209.522376 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000193 1788622209.639986 write 0x03cdd8 thru 0x03ffff (0x3228 bytes) 000195 1788622209.651417 trunc from 0x040000 to 0x01f1f8 (0x20e07 bytes) 000196 1788622209.669398 write 0x00e20a thru 0x01b7a3 (0xd59a bytes) 000197 1788622209.678358 mapread 0x0181e6 thru 0x01f1f8 (0x7013 bytes) 000198 1788622209.690569 trunc from 0x01f1f9 to 0x01a72b (0x4acd bytes) 000199 1788622209.703824 read 0x00548b thru 0x008dea (0x3960 bytes) 000202 1788622209.720975 mapread 0x0007b3 thru 0x0052fa (0x4b48 bytes) 000203 1788622209.734197 MIRROR_EXTEND to 2 mirrors 000205 1788622210.345228 MIRROR_EXTEND to 3 mirrors 000207 1788622211.072054 MIRROR_RESYNC 3 mirrors 000208 1788622211.702900 mapwrite 0x00f785 thru 0x01e952 (0xf1ce bytes) 000209 1788622212.021589 mapwrite 0x021c9c thru 0x02a843 (0x8ba8 bytes) 000210 1788622212.131492 read 0x0275d9 thru 0x02a843 (0x326b bytes) 000211 1788622212.132246 write 0x0100eb thru 0x016d40 (0x6c56 bytes) 000212 1788622212.138996 MIRROR_RESYNC 3 mirrors 000213 1788622212.909477 trunc from 0x02a844 to 0x03d75a (0x12f17 bytes) 000214 1788622212.997031 trunc from 0x03d75b to 0x03e73f (0xfe5 bytes) 000216 1788622213.020243 read 0x00ff99 thru 0x016c6e (0x6cd6 bytes) 000219 1788622213.029389 mapwrite 0x019b29 thru 0x024d55 (0xb22d bytes) 000220 1788622213.094064 trunc from 0x03e740 to 0x00083c (0x3df03 bytes) 000222 1788622213.117030 trunc from 0x00083d to 0x0033c8 (0x2b8c bytes) 000223 1788622213.138373 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33c9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c9): Mirror 2: 0xb6d8508 (stale) Mirror 3: 0xb6d8508 (stale) CRC-32 checksum value for chunk [0, 0x33c9): Mirror 1: 0xa88a9ce5 Mirror 4: 0xa88a9ce5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c9: skipped 000224 1788622213.876529 MIRROR_RESYNC 4 mirrors 000226 1788622214.525965 read 0x00154e thru 0x0033c8 (0x1e7b bytes) 000227 1788622214.532868 read 0x002827 thru 0x0033c8 (0xba2 bytes) 000228 1788622214.533937 write 0x02d541 thru 0x0301c8 (0x2c88 bytes) 000231 1788622214.605171 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301c9): Mirror 2: 0xc133fbc8 (stale) Mirror 3: 0xc133fbc8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301c9): Mirror 2: 0xc133fbc8 (stale) Mirror 3: 0xc133fbc8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301c9: skipped 000232 1788622214.902430 read 0x00e9c3 thru 0x018916 (0x9f54 bytes) 000234 1788622214.913404 mapread 0x01d88a thru 0x022176 (0x48ed bytes) 000235 1788622214.932697 write 0x03d86e thru 0x03ffff (0x2792 bytes) 000236 1788622214.933696 read 0x00778d thru 0x0103d7 (0x8c4b bytes) 000237 1788622214.943321 write 0x027381 thru 0x02a5df (0x325f bytes) 000238 1788622214.944485 read 0x011622 thru 0x020ef7 (0xf8d6 bytes) 000241 1788622214.947529 trunc from 0x040000 to 0x03de88 (0x2177 bytes) 000242 1788622214.990290 trunc from 0x03de89 to 0x038dc4 (0x50c4 bytes) 000243 1788622215.008761 write 0x013d5a thru 0x019062 (0x5309 bytes) 000244 1788622215.009940 write 0x03c4dc thru 0x03e37c (0x1ea1 bytes) 000246 1788622215.016958 MIRROR_RESYNC 3 mirrors 000250 1788622216.345578 read 0x012aee thru 0x01df37 (0xb44a bytes) 000251 1788622216.359004 mapwrite 0x0370f9 thru 0x03ffff (0x8f07 bytes) 000252 1788622216.506755 mapread 0x010027 thru 0x019184 (0x915e bytes) 000253 1788622216.524637 trunc from 0x040000 to 0x00c15a (0x33ea5 bytes) 000254 1788622216.542515 trunc from 0x00c15b to 0x02ffc9 (0x23e6f bytes) 000255 1788622216.571437 MIRROR_RESYNC 3 mirrors 000256 1788622217.251632 write 0x036342 thru 0x03f324 (0x8fe3 bytes) 000258 1788622217.307545 trunc from 0x03f325 to 0x0380c8 (0x725c bytes) 000259 1788622217.335271 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x380c9): Mirror 2: 0xf722f970 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380c9): Mirror 2: 0xf722f970 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380c9: skipped 000261 1788622217.572307 mapwrite 0x00542f thru 0x012a27 (0xd5f9 bytes) 000263 1788622217.669038 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000264 1788622217.785319 read 0x021fd7 thru 0x028a02 (0x6a2c bytes) 000266 1788622217.795387 mapread 0x005bde thru 0x007e5f (0x2282 bytes) 000271 1788622217.815153 read 0x0094f9 thru 0x012c71 (0x9779 bytes) 000272 1788622217.827088 MIRROR_EXTEND to 2 mirrors 000276 1788622218.584334 mapwrite 0x01f229 thru 0x028eeb (0x9cc3 bytes) 000278 1788622218.751716 trunc from 0x0380c9 to 0x014822 (0x238a6 bytes) 000280 1788622218.783766 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000283 1788622218.931474 read 0x00c6c5 thru 0x0123c3 (0x5cff bytes) 000284 1788622218.939582 mapread 0x009cfc thru 0x00f7b7 (0x5abc bytes) 000288 1788622218.960569 MIRROR_EXTEND to 2 mirrors 000290 1788622219.861167 read 0x012b97 thru 0x014822 (0x1c8c bytes) 000291 1788622219.872165 mapread 0x0106ec thru 0x012fee (0x2903 bytes) 000293 1788622219.899285 trunc from 0x014823 to 0x032332 (0x1db10 bytes) 000294 1788622219.961425 write 0x010f25 thru 0x01d113 (0xc1ef bytes) 000296 1788622219.973599 read 0x011d76 thru 0x014bf2 (0x2e7d bytes) 000297 1788622219.974910 write 0x01ca6f thru 0x024ad7 (0x8069 bytes) 000298 1788622219.985987 mapread 0x005437 thru 0x00d2df (0x7ea9 bytes) 000299 1788622219.987406 mapread 0x02d6b4 thru 0x02e9d5 (0x1322 bytes) 000301 1788622219.999928 MIRROR_RESYNC 2 mirrors 000302 1788622220.575631 read 0x0223a5 thru 0x027a92 (0x56ee bytes) 000303 1788622220.589259 MIRROR_RESYNC 2 mirrors 000304 1788622220.842215 mapwrite 0x00fd69 thru 0x01947c (0x9714 bytes) 000307 1788622221.015438 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32333): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32333): Mirror 2: 0xbaddf0e2 (stale) CRC-32 checksum value for chunk [0, 0x32333): Mirror 1: 0x904acf3b Mirror 3: 0x904acf3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32333: skipped 000308 1788622221.983387 trunc from 0x032333 to 0x01bdae (0x16584 bytes) 000309 1788622222.135461 mapwrite 0x02cf93 thru 0x03c49e (0xf50c bytes) 000310 1788622222.259450 read 0x038865 thru 0x03c49e (0x3c3a bytes) 000311 1788622222.263990 MIRROR_RESYNC 3 mirrors 000315 1788622223.019979 mapwrite 0x01f0b9 thru 0x01fcef (0xc37 bytes) 000316 1788622223.147125 mapwrite 0x00d6fc thru 0x011718 (0x401d bytes) 000317 1788622223.189595 read 0x00459a thru 0x009075 (0x4adc bytes) 000318 1788622223.194779 trunc from 0x03c49f to 0x02d9b5 (0xeae9 bytes) 000322 1788622223.237850 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d9b6): Mirror 3: 0xc11c75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d9b6): Mirror 3: 0xc11c75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d9b6: skipped 000325 1788622223.471805 trunc from 0x02d9b6 to 0x020b65 (0xce50 bytes) 000326 1788622223.486935 mapread 0x0000a0 thru 0x001fc5 (0x1f26 bytes) 000327 1788622223.496774 read 0x01269e thru 0x018b58 (0x64bb bytes) 000329 1788622223.504394 mapwrite 0x01abd5 thru 0x01bf0d (0x1339 bytes) 000331 1788622223.546775 write 0x01a5be thru 0x027fdc (0xda1f bytes) 000332 1788622223.549170 read 0x000629 thru 0x00b0eb (0xaac3 bytes) 000334 1788622223.560859 MIRROR_RESYNC 2 mirrors 000335 1788622224.050521 mapwrite 0x00b074 thru 0x012712 (0x769f bytes) 000341 1788622224.182853 mapwrite 0x034cd3 thru 0x03d75d (0x8a8b bytes) 000343 1788622224.277802 MIRROR_RESYNC 2 mirrors 000344 1788622224.990426 mapwrite 0x003e97 thru 0x00cb88 (0x8cf2 bytes) 000345 1788622225.172872 MIRROR_RESYNC 2 mirrors 000348 1788622225.821345 write 0x03f090 thru 0x03ffff (0xf70 bytes) 000349 1788622225.868556 write 0x003684 thru 0x0075f1 (0x3f6e bytes) 000350 1788622225.877804 mapread 0x0198f7 thru 0x01b5ef (0x1cf9 bytes) 000352 1788622225.895139 read 0x0269c2 thru 0x027d37 (0x1376 bytes) 000353 1788622225.900282 mapread 0x00525a thru 0x00b670 (0x6417 bytes) 000356 1788622225.910289 trunc from 0x040000 to 0x00abbd (0x35442 bytes) 000358 1788622225.928420 read 0x002a2a thru 0x00abbd (0x8194 bytes) 000360 1788622225.930292 trunc from 0x00abbe to 0x01f84d (0x14c90 bytes) 000362 1788622225.949925 mapread 0x015588 thru 0x01d7b9 (0x8232 bytes) 000363 1788622225.959467 read 0x007e89 thru 0x0159aa (0xdb22 bytes) 000364 1788622225.960140 trunc from 0x01f84e to 0x038e75 (0x19628 bytes) 000369 1788622225.980933 mapread 0x005837 thru 0x0126c7 (0xce91 bytes) 000372 1788622225.984301 write 0x002c02 thru 0x00636d (0x376c bytes) 000373 1788622225.985555 mapread 0x00c6b4 thru 0x011441 (0x4d8e bytes) 000379 1788622225.993381 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38e76): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38e76): Mirror 3: 0xfa4301ef (stale) CRC-32 checksum value for chunk [0, 0x38e76): Mirror 1: 0x270e8af Mirror 4: 0x270e8af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38e76: skipped 000383 1788622226.771894 mapwrite 0x006bfa thru 0x00c807 (0x5c0e bytes) 000385 1788622226.920330 write 0x02d47c thru 0x03c26e (0xedf3 bytes) 000386 1788622226.927175 read 0x001ab2 thru 0x007b22 (0x6071 bytes) 000387 1788622226.927820 trunc from 0x03c26f to 0x03e492 (0x2224 bytes) 000388 1788622226.948273 mapwrite 0x037eb3 thru 0x03ffff (0x814d bytes) 000392 1788622227.042344 mapread 0x01970f thru 0x02571e (0xc010 bytes) 000393 1788622227.065967 write 0x03831d thru 0x03ffff (0x7ce3 bytes) 000394 1788622227.067841 write 0x00b4f1 thru 0x013b66 (0x8676 bytes) 000395 1788622227.072491 write 0x0147ee thru 0x0178b1 (0x30c4 bytes) 000398 1788622227.076237 mapwrite 0x0062c4 thru 0x00c514 (0x6251 bytes) 000399 1788622227.146829 read 0x02549a thru 0x030573 (0xb0da bytes) 000400 1788622227.157435 mapread 0x01e240 thru 0x02db7e (0xf93f bytes) 000402 1788622227.158853 mapread 0x02743c thru 0x02ac13 (0x37d8 bytes) 000403 1788622227.159601 write 0x0230ff thru 0x02bf6a (0x8e6c bytes) 000404 1788622227.161839 mapread 0x02e50a thru 0x03bad7 (0xd5ce bytes) 000405 1788622227.163162 write 0x0008b6 thru 0x00ebfa (0xe345 bytes) 000407 1788622227.165403 mapwrite 0x00b8ca thru 0x019ee2 (0xe619 bytes) 000409 1788622227.219682 trunc from 0x040000 to 0x00f60c (0x309f3 bytes) 000410 1788622227.244092 mapread 0x000cde thru 0x003253 (0x2576 bytes) 000411 1788622227.245702 mapwrite 0x016275 thru 0x0254c8 (0xf254 bytes) 000413 1788622227.422432 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x254c9): Mirror 3: 0x76e58371 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x254c9): Mirror 3: 0x76e58371 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x254c9: skipped 000415 1788622227.692097 mapwrite 0x004428 thru 0x008ca7 (0x4880 bytes) 000417 1788622227.756344 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x254c9): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x254c9): Mirror 3: 0x76e58371 (stale) CRC-32 checksum value for chunk [0, 0x254c9): Mirror 1: 0x2bd68a10 Mirror 4: 0x2bd68a10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x254c9: skipped 000418 1788622228.616145 read 0x012cda thru 0x021339 (0xe660 bytes) 000424 1788622228.672310 write 0x007722 thru 0x0083e8 (0xcc7 bytes) 000425 1788622228.745331 write 0x039d7d thru 0x03ffff (0x6283 bytes) 000427 1788622228.752744 read 0x002b14 thru 0x00fb8b (0xd078 bytes) 000429 1788622228.775120 write 0x014b2c thru 0x01672b (0x1c00 bytes) 000430 1788622228.776023 mapread 0x022fdd thru 0x029129 (0x614d bytes) 000431 1788622228.776889 mapwrite 0x001816 thru 0x00da6b (0xc256 bytes) 000432 1788622228.936509 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xdcba4620 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xdcba4620 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000434 1788622229.194450 trunc from 0x040000 to 0x01c60d (0x239f2 bytes) 000437 1788622229.209544 mapread 0x017b97 thru 0x01c60d (0x4a77 bytes) 000438 1788622229.229851 MIRROR_RESYNC 2 mirrors 000439 1788622229.903733 mapread 0x007599 thru 0x0156e2 (0xe14a bytes) 000441 1788622229.931492 mapwrite 0x004fe7 thru 0x012fbd (0xdfd7 bytes) 000443 1788622230.057881 mapwrite 0x039df1 thru 0x03ffff (0x620f bytes) 000445 1788622230.158697 read 0x01a751 thru 0x02714e (0xc9fe bytes) 000447 1788622230.171286 mapread 0x0208aa thru 0x02c97c (0xc0d3 bytes) 000449 1788622230.189809 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x83c2a7c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2d69aef6 Mirror 5: 0x2d69aef6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000453 1788622231.550968 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x83c2a7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x83c2a7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000455 1788622231.975220 mapread 0x00277d thru 0x0045ca (0x1e4e bytes) 000456 1788622232.004031 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x83c2a7c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2d69aef6 Mirror 5: 0x2d69aef6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000458 1788622232.925627 write 0x039aac thru 0x03e3b9 (0x490e bytes) 000461 1788622233.007852 trunc from 0x040000 to 0x011398 (0x2ec67 bytes) 000465 1788622233.029435 mapwrite 0x00d12a thru 0x012e40 (0x5d17 bytes) 000466 1788622233.099089 write 0x013e7b thru 0x021e72 (0xdff8 bytes) 000467 1788622233.101477 mapread 0x01d30a thru 0x021e72 (0x4b69 bytes) 000468 1788622233.102865 write 0x02d61c thru 0x02d891 (0x276 bytes) 000469 1788622233.110460 MIRROR_RESYNC 3 mirrors 000470 1788622233.947353 trunc from 0x02d892 to 0x033330 (0x5a9f bytes) 000471 1788622234.045264 mapread 0x01f30a thru 0x020da2 (0x1a99 bytes) 000472 1788622234.051127 mapread 0x0056bc thru 0x007033 (0x1978 bytes) 000473 1788622234.057007 write 0x007416 thru 0x009d50 (0x293b bytes) 000474 1788622234.057608 write 0x0039a6 thru 0x00a189 (0x67e4 bytes) 000475 1788622234.058397 mapwrite 0x0132ff thru 0x02313d (0xfe3f bytes) 000478 1788622234.106358 trunc from 0x033331 to 0x005bc9 (0x2d767 bytes) 000479 1788622234.123129 write 0x01ee1c thru 0x02d66b (0xe850 bytes) 000482 1788622234.131118 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d66c): Mirror 5: 0x22eb550 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d66c): Mirror 5: 0x22eb550 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d66c: skipped 000483 1788622234.408314 MIRROR_RESYNC 2 mirrors 000484 1788622235.069814 mapread 0x021584 thru 0x027b89 (0x6606 bytes) 000485 1788622235.093337 mapwrite 0x01ef27 thru 0x02ac53 (0xbd2d bytes) 000486 1788622235.234383 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d66c): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d66c): Mirror 5: 0x9c68549e (stale) CRC-32 checksum value for chunk [0, 0x2d66c): Mirror 1: 0xe5fb2def Mirror 6: 0xe5fb2def lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d66c: skipped 000487 1788622236.266332 mapread 0x019202 thru 0x029182 (0xff81 bytes) 000488 1788622236.329884 mapread 0x018c76 thru 0x01ae4c (0x21d7 bytes) 000490 1788622236.367536 mapwrite 0x02d17f thru 0x036579 (0x93fb bytes) 000493 1788622236.642145 write 0x024012 thru 0x0292af (0x529e bytes) 000494 1788622236.671931 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3657a): Mirror 5: 0x1db9f00f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3657a): Mirror 5: 0x1db9f00f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3657a: skipped 000496 1788622237.122483 mapwrite 0x00393e thru 0x00d016 (0x96d9 bytes) 000498 1788622237.251496 read 0x00347c thru 0x00735e (0x3ee3 bytes) 000499 1788622237.253224 trunc from 0x03657a to 0x01c4bc (0x1a0bd bytes) 000500 1788622237.275295 read 0x00028a thru 0x00d53a (0xd2b1 bytes) 000502 1788622237.303253 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000508 1788622237.485165 write 0x002af6 thru 0x00ab33 (0x803e bytes) 000509 1788622237.503683 write 0x03b7c2 thru 0x03ffff (0x483e bytes) 000511 1788622237.505819 read 0x024676 thru 0x027ff1 (0x397c bytes) 000512 1788622237.518473 read 0x01cf90 thru 0x02cc45 (0xfcb6 bytes) 000514 1788622237.536987 write 0x030337 thru 0x038035 (0x7cff bytes) 000517 1788622237.538848 mapread 0x02e881 thru 0x03b8b6 (0xd036 bytes) 000519 1788622237.544465 mapwrite 0x03519e thru 0x035737 (0x59a bytes) 000520 1788622237.647713 MIRROR_EXTEND to 2 mirrors 000521 1788622239.090505 write 0x03a88f thru 0x03ffff (0x5771 bytes) 000522 1788622239.190890 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000524 1788622239.360872 write 0x03e308 thru 0x03ffff (0x1cf8 bytes) 000525 1788622239.374615 MIRROR_EXTEND to 2 mirrors 000526 1788622240.763441 trunc from 0x040000 to 0x028cc2 (0x1733d bytes) 000529 1788622240.866290 read 0x00d194 thru 0x013dca (0x6c37 bytes) 000530 1788622240.877538 write 0x03a2bf thru 0x03ffff (0x5d41 bytes) 000531 1788622240.880411 mapwrite 0x0121c3 thru 0x01e9b8 (0xc7f6 bytes) 000533 1788622241.011010 trunc from 0x040000 to 0x00446d (0x3bb92 bytes) 000538 1788622241.132357 mapread 0x000baa thru 0x001d59 (0x11b0 bytes) 000542 1788622241.170778 write 0x011b86 thru 0x01b97c (0x9df7 bytes) 000544 1788622241.184700 mapread 0x00f84f thru 0x01748e (0x7c40 bytes) 000545 1788622241.229038 mapwrite 0x03f9ae thru 0x03ffff (0x652 bytes) 000547 1788622241.459774 MIRROR_RESYNC 2 mirrors 000548 1788622243.313483 MIRROR_RESYNC 2 mirrors 000549 1788622244.109822 trunc from 0x040000 to 0x00c8b0 (0x3374f bytes) 000551 1788622244.187773 mapread 0x001172 thru 0x00c8b0 (0xb73f bytes) 000552 1788622244.212139 trunc from 0x00c8b1 to 0x037266 (0x2a9b6 bytes) 000553 1788622244.259949 read 0x020a12 thru 0x02ca5f (0xc04e bytes) 000557 1788622244.278240 read 0x013441 thru 0x01f423 (0xbfe3 bytes) 000558 1788622244.299352 write 0x03ab53 thru 0x03ffff (0x54ad bytes) 000559 1788622244.302023 mapwrite 0x013163 thru 0x02150c (0xe3aa bytes) 000561 1788622244.397048 mapwrite 0x03c625 thru 0x03ffff (0x39db bytes) 000562 1788622244.457621 write 0x03b092 thru 0x03ffff (0x4f6e bytes) 000563 1788622244.458834 trunc from 0x040000 to 0x01f44c (0x20bb3 bytes) 000567 1788622244.489430 mapread 0x004d4b thru 0x012445 (0xd6fb bytes) 000568 1788622244.498128 MIRROR_RESYNC 2 mirrors 000569 1788622245.613319 mapread 0x00152c thru 0x00d968 (0xc43d bytes) 000570 1788622245.649433 write 0x01fad9 thru 0x0288db (0x8e03 bytes) 000573 1788622245.743522 write 0x01734b thru 0x01cd96 (0x5a4c bytes) 000574 1788622245.759304 mapread 0x0166bd thru 0x0256b4 (0xeff8 bytes) 000577 1788622245.800607 trunc from 0x0288dc to 0x00dab0 (0x1ae2b bytes) 000580 1788622245.843636 mapread 0x007966 thru 0x00c45e (0x4af9 bytes) 000581 1788622245.852261 read 0x0059b9 thru 0x00b3dd (0x5a25 bytes) 000582 1788622245.858107 trunc from 0x00dab1 to 0x031d3c (0x2428c bytes) 000584 1788622245.871876 mapwrite 0x02dd84 thru 0x035102 (0x737f bytes) 000586 1788622245.976430 read 0x030605 thru 0x035102 (0x4afe bytes) 000588 1788622245.978298 mapwrite 0x021c50 thru 0x029044 (0x73f5 bytes) 000589 1788622246.032027 trunc from 0x035103 to 0x00a70c (0x2a9f6 bytes) 000590 1788622246.064983 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xa70d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa70d): Mirror 2: 0x19c59f71 (stale) CRC-32 checksum value for chunk [0, 0xa70d): Mirror 1: 0x19c59f71 Mirror 3: 0x19c59f71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa70d: skipped 000593 1788622247.310302 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000595 1788622247.838196 trunc from 0x00a70d to 0x0238fd (0x191f1 bytes) 000598 1788622248.066171 write 0x0381c9 thru 0x03ffff (0x7e37 bytes) 000599 1788622248.072027 mapread 0x031acd thru 0x03cdc3 (0xb2f7 bytes) 000600 1788622248.110035 read 0x036f2a thru 0x03ffff (0x90d6 bytes) 000601 1788622248.110762 mapwrite 0x02bb4a thru 0x02c5f5 (0xaac bytes) 000602 1788622248.210627 mapread 0x028aab thru 0x036787 (0xdcdd bytes) 000604 1788622248.212164 trunc from 0x040000 to 0x01806c (0x27f93 bytes) 000606 1788622248.257155 MIRROR_RESYNC 2 mirrors 000611 1788622248.841983 trunc from 0x01806d to 0x010d36 (0x7336 bytes) 000613 1788622248.915491 trunc from 0x010d37 to 0x00df82 (0x2db4 bytes) 000614 1788622248.973746 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000615 1788622249.087121 trunc from 0x00df83 to 0x011dfb (0x3e79 bytes) 000616 1788622249.103429 mapread 0x00c394 thru 0x00f2bb (0x2f28 bytes) 000620 1788622249.124164 read 0x001498 thru 0x002061 (0xbca bytes) 000622 1788622249.132575 write 0x02db85 thru 0x03daee (0xff6a bytes) 000624 1788622249.135708 mapread 0x0149c0 thru 0x024463 (0xfaa4 bytes) 000628 1788622249.151034 trunc from 0x03daef to 0x01375a (0x2a394 bytes) 000631 1788622249.172056 write 0x00e48c thru 0x01b4cd (0xd042 bytes) 000632 1788622249.173993 write 0x012c4c thru 0x022b1a (0xfecf bytes) 000633 1788622249.176612 mapwrite 0x02ef68 thru 0x0349a2 (0x5a3b bytes) 000635 1788622249.243479 MIRROR_EXTEND to 2 mirrors 000636 1788622250.075126 read 0x01e843 thru 0x02431b (0x5ad9 bytes) 000637 1788622250.092087 read 0x02ac51 thru 0x031f9d (0x734d bytes) 000639 1788622250.124215 mapwrite 0x0110cc thru 0x018762 (0x7697 bytes) 000640 1788622250.329440 mapwrite 0x03328a thru 0x035b3e (0x28b5 bytes) 000641 1788622250.502292 read 0x00e717 thru 0x017a7a (0x9364 bytes) 000642 1788622250.524633 read 0x00656d thru 0x008bf9 (0x268d bytes) 000643 1788622250.525333 mapwrite 0x0053bf thru 0x0153b2 (0xfff4 bytes) 000645 1788622250.633736 read 0x00bc85 thru 0x01aff2 (0xf36e bytes) 000646 1788622250.634380 write 0x036501 thru 0x03ffff (0x9aff bytes) 000649 1788622250.645021 trunc from 0x040000 to 0x03b987 (0x4678 bytes) 000654 1788622250.705114 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b988): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b988): Mirror 2: 0x7d8540ef (stale) CRC-32 checksum value for chunk [0, 0x3b988): Mirror 1: 0x90449357 Mirror 3: 0x90449357 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b988: skipped 000655 1788622251.758420 read 0x02d6e4 thru 0x0375d9 (0x9ef6 bytes) 000658 1788622251.808559 mapwrite 0x031d28 thru 0x03b77b (0x9a54 bytes) 000659 1788622252.116432 mapread 0x031409 thru 0x03b987 (0xa57f bytes) 000662 1788622252.127501 mapread 0x005027 thru 0x00d2b3 (0x828d bytes) 000663 1788622252.151901 mapwrite 0x00656e thru 0x00e593 (0x8026 bytes) 000664 1788622252.273275 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b988): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b988): Mirror 2: 0x7d8540ef (stale) Mirror 3: 0x90449357 (stale) CRC-32 checksum value for chunk [0, 0x3b988): Mirror 1: 0xa0c07bdf Mirror 4: 0xa0c07bdf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b988: skipped 000666 1788622253.471081 mapread 0x008341 thru 0x00e173 (0x5e33 bytes) 000668 1788622253.513098 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b988): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b988): Mirror 2: 0x7d8540ef (stale) Mirror 3: 0x90449357 (stale) CRC-32 checksum value for chunk [0, 0x3b988): Mirror 1: 0xa0c07bdf Mirror 4: 0xa0c07bdf Mirror 5: 0xa0c07bdf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b988: skipped 000669 1788622254.486043 write 0x011c25 thru 0x01eb49 (0xcf25 bytes) 000671 1788622254.637481 mapread 0x00c371 thru 0x0175c4 (0xb254 bytes) 000672 1788622254.651198 mapread 0x0090bf thru 0x00916e (0xb0 bytes) 000673 1788622254.651992 mapread 0x0288cf thru 0x0358e7 (0xd019 bytes) 000677 1788622254.676133 MIRROR_RESYNC 5 mirrors 000679 1788622255.799853 MIRROR_EXTEND to 6 mirrors 000680 1788622256.493185 mapwrite 0x00d490 thru 0x013638 (0x61a9 bytes) 000682 1788622256.691511 mapread 0x036439 thru 0x03b987 (0x554f bytes) 000684 1788622256.702231 mapread 0x010a8a thru 0x01f92b (0xeea2 bytes) 000686 1788622256.712522 trunc from 0x03b988 to 0x01adae (0x20bd9 bytes) 000687 1788622256.732198 mapwrite 0x00a9eb thru 0x00b142 (0x758 bytes) 000688 1788622256.764334 read 0x011065 thru 0x012d7e (0x1d1a bytes) 000689 1788622256.764956 read 0x006b1d thru 0x009b5e (0x3042 bytes) 000690 1788622256.776815 MIRROR_RESYNC 6 mirrors 000694 1788622257.945583 write 0x02739d thru 0x02a363 (0x2fc7 bytes) 000695 1788622258.060392 mapread 0x01f112 thru 0x02611e (0x700d bytes) 000700 1788622258.096625 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a364): Mirror 2: 0x4d948b49 (stale) Mirror 3: 0x4d948b49 (stale) Mirror 4: 0x4d948b49 (stale) Mirror 5: 0x4d948b49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a364): Mirror 2: 0x4d948b49 (stale) Mirror 3: 0x4d948b49 (stale) Mirror 4: 0x4d948b49 (stale) Mirror 5: 0x4d948b49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a364: skipped 000702 1788622258.762040 mapwrite 0x01497d thru 0x022468 (0xdaec bytes) 000704 1788622258.880403 read 0x01075d thru 0x011908 (0x11ac bytes) 000705 1788622258.881338 write 0x00c96d thru 0x0199eb (0xd07f bytes) 000706 1788622258.894235 read 0x029428 thru 0x02a363 (0xf3c bytes) 000707 1788622258.909275 write 0x01bfc8 thru 0x02acae (0xece7 bytes) 000709 1788622258.912452 write 0x010ffe thru 0x0111b4 (0x1b7 bytes) 000710 1788622258.914468 write 0x020e91 thru 0x021ca2 (0xe12 bytes) 000714 1788622258.917605 read 0x01cb75 thru 0x027e6a (0xb2f6 bytes) 000716 1788622258.920263 write 0x0215dc thru 0x02f626 (0xe04b bytes) 000717 1788622258.938211 MIRROR_RESYNC 5 mirrors 000718 1788622260.063412 read 0x02bbd9 thru 0x02d931 (0x1d59 bytes) 000719 1788622260.076568 MIRROR_EXTEND to 6 mirrors 000720 1788622261.195416 MIRROR_RESYNC 6 mirrors 000721 1788622261.675470 MIRROR_RESYNC 6 mirrors 000722 1788622262.065506 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000723 1788622262.480126 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000727 1788622262.828386 mapread 0x0033e0 thru 0x010d0a (0xd92b bytes) 000728 1788622262.858070 mapwrite 0x00c268 thru 0x011882 (0x561b bytes) 000736 1788622263.111950 trunc from 0x02f627 to 0x007ac4 (0x27b62 bytes) 000738 1788622263.155032 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7ac5): Mirror 5: 0x11537e1b (stale) Mirror 6: 0x11537e1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7ac5): Mirror 5: 0x11537e1b (stale) Mirror 6: 0x11537e1b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7ac5: skipped 000739 1788622263.440362 mapwrite 0x00079d thru 0x004d25 (0x4589 bytes) 000741 1788622263.494516 mapread 0x0018d1 thru 0x007ac4 (0x61f4 bytes) 000743 1788622263.496282 mapread 0x0039f8 thru 0x007ac4 (0x40cd bytes) 000744 1788622263.496980 read 0x004f93 thru 0x007ac4 (0x2b32 bytes) 000746 1788622263.498233 mapread 0x0067e2 thru 0x007ac4 (0x12e3 bytes) 000747 1788622263.499026 mapread 0x001794 thru 0x007ac4 (0x6331 bytes) 000748 1788622263.500250 trunc from 0x007ac5 to 0x00ebd0 (0x710c bytes) 000749 1788622263.516944 MIRROR_RESYNC 3 mirrors 000750 1788622264.253966 mapread 0x007235 thru 0x00b980 (0x474c bytes) 000751 1788622264.290584 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000752 1788622264.535810 mapwrite 0x03cbd0 thru 0x03ffb4 (0x33e5 bytes) 000753 1788622264.677267 mapwrite 0x00770d thru 0x00b057 (0x394b bytes) 000755 1788622264.731297 write 0x01229c thru 0x0184b7 (0x621c bytes) 000760 1788622264.744305 mapread 0x0086ca thru 0x014052 (0xb989 bytes) 000761 1788622264.752239 write 0x004408 thru 0x00aa47 (0x6640 bytes) 000764 1788622264.755293 mapwrite 0x001a64 thru 0x007d4d (0x62ea bytes) 000766 1788622264.804198 write 0x0216cd thru 0x028bf2 (0x7526 bytes) 000767 1788622264.814344 write 0x003c9c thru 0x009285 (0x55ea bytes) 000768 1788622264.815956 read 0x03decd thru 0x03ffb4 (0x20e8 bytes) 000769 1788622264.817438 mapread 0x039cd0 thru 0x03ce1a (0x314b bytes) 000770 1788622264.818821 mapwrite 0x032ad2 thru 0x03ea2a (0xbf59 bytes) 000774 1788622264.878184 mapwrite 0x03218e thru 0x036adc (0x494f bytes) 000775 1788622264.921488 mapread 0x02f536 thru 0x039696 (0xa161 bytes) 000776 1788622264.923115 write 0x02c949 thru 0x02d452 (0xb0a bytes) 000779 1788622264.927973 trunc from 0x03ffb5 to 0x02d0b2 (0x12f02 bytes) 000785 1788622264.962096 MIRROR_RESYNC 2 mirrors 000786 1788622265.728705 mapwrite 0x01ed16 thru 0x029cb6 (0xafa1 bytes) 000787 1788622265.903733 trunc from 0x02d0b3 to 0x03670d (0x965b bytes) 000788 1788622265.923984 write 0x001281 thru 0x002114 (0xe94 bytes) 000790 1788622265.937290 read 0x00206a thru 0x00a9bc (0x8953 bytes) 000792 1788622265.946686 mapread 0x02d806 thru 0x03670d (0x8f08 bytes) 000795 1788622265.958444 write 0x028a35 thru 0x02befb (0x34c7 bytes) 000797 1788622265.962473 trunc from 0x03670e to 0x01c240 (0x1a4cd bytes) 000799 1788622266.002409 trunc from 0x01c241 to 0x0080d0 (0x14170 bytes) 000800 1788622266.021447 write 0x0279e7 thru 0x032bd6 (0xb1f0 bytes) 000801 1788622266.028807 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x32bd7): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32bd7): Mirror 5: 0x4bdcd7f8 (stale) CRC-32 checksum value for chunk [0, 0x32bd7): Mirror 1: 0x6059980a Mirror 6: 0x6059980a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32bd7: skipped 000803 1788622267.235517 write 0x019b0b thru 0x01e569 (0x4a5f bytes) 000805 1788622267.329586 trunc from 0x032bd7 to 0x03dadb (0xaf05 bytes) 000806 1788622267.347429 trunc from 0x03dadc to 0x025f3c (0x17b9f bytes) 000808 1788622267.372182 mapread 0x00c9fa thru 0x00fc03 (0x320a bytes) 000812 1788622267.398356 trunc from 0x025f3d to 0x02c2ff (0x63c3 bytes) 000813 1788622267.433469 read 0x00c40a thru 0x012b61 (0x6758 bytes) 000814 1788622267.440137 trunc from 0x02c300 to 0x017b7f (0x14780 bytes) 000824 1788622267.474476 mapread 0x002a76 thru 0x00aca1 (0x822c bytes) 000825 1788622267.506479 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17b80): Mirror 5: 0x214c305 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17b80): Mirror 5: 0x214c305 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17b80: skipped 000827 1788622267.871487 trunc from 0x017b80 to 0x03a795 (0x22c16 bytes) 000829 1788622267.898611 MIRROR_RESYNC 2 mirrors 000831 1788622268.520869 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000832 1788622268.644228 mapwrite 0x039e24 thru 0x03e800 (0x49dd bytes) 000833 1788622268.725702 write 0x03b29b thru 0x03ffff (0x4d65 bytes) 000834 1788622268.726610 read 0x019522 thru 0x0208ee (0x73cd bytes) 000835 1788622268.733157 mapwrite 0x03993e thru 0x03af68 (0x162b bytes) 000837 1788622268.770012 mapread 0x024d1a thru 0x02e2ac (0x9593 bytes) 000839 1788622268.783375 mapwrite 0x011088 thru 0x017112 (0x608b bytes) 000840 1788622268.833756 trunc from 0x040000 to 0x03940e (0x6bf1 bytes) 000841 1788622268.851836 read 0x00b9d0 thru 0x00f4aa (0x3adb bytes) 000842 1788622268.852476 trunc from 0x03940f to 0x0126fb (0x26d13 bytes) 000844 1788622268.868246 trunc from 0x0126fc to 0x025df1 (0x136f6 bytes) 000847 1788622268.884396 mapwrite 0x007dd2 thru 0x00dc0d (0x5e3c bytes) 000850 1788622268.924354 read 0x0138ab thru 0x01f93a (0xc090 bytes) 000853 1788622268.925937 write 0x03c145 thru 0x03dd9a (0x1c56 bytes) 000855 1788622268.927408 mapread 0x02a49b thru 0x0365d1 (0xc137 bytes) 000857 1788622268.934820 trunc from 0x03dd9b to 0x00b0cf (0x32ccb bytes) 000861 1788622268.948798 read 0x004970 thru 0x005c87 (0x1318 bytes) 000862 1788622268.951358 MIRROR_EXTEND to 2 mirrors 000863 1788622269.480983 mapread 0x002872 thru 0x00b0cf (0x885e bytes) 000868 1788622269.648872 mapwrite 0x002230 thru 0x0076c7 (0x5498 bytes) 000869 1788622269.984723 mapwrite 0x02825d thru 0x035f8b (0xdd2f bytes) 000870 1788622270.115376 read 0x022e66 thru 0x02e4cb (0xb666 bytes) 000872 1788622270.116536 trunc from 0x035f8c to 0x03e624 (0x8699 bytes) 000873 1788622270.146451 read 0x01913b thru 0x0262e8 (0xd1ae bytes) 000874 1788622270.159271 write 0x023431 thru 0x02a7eb (0x73bb bytes) 000875 1788622270.164542 trunc from 0x03e625 to 0x01c23f (0x223e5 bytes) 000876 1788622270.230610 mapread 0x00d372 thru 0x015b6d (0x87fc bytes) 000877 1788622270.239447 mapwrite 0x01544b thru 0x01da74 (0x862a bytes) 000878 1788622270.389577 mapwrite 0x029a60 thru 0x034b52 (0xb0f3 bytes) 000879 1788622270.627930 trunc from 0x034b53 to 0x037bb3 (0x3061 bytes) 000880 1788622270.677251 read 0x02d0e8 thru 0x02d7c0 (0x6d9 bytes) 000882 1788622270.678519 read 0x02a300 thru 0x02a961 (0x662 bytes) 000886 1788622270.683952 mapread 0x02ef19 thru 0x037bb3 (0x8c9b bytes) 000888 1788622270.696263 mapwrite 0x018519 thru 0x0267ec (0xe2d4 bytes) 000890 1788622270.765860 read 0x02aa07 thru 0x02b447 (0xa41 bytes) 000891 1788622270.766547 read 0x021826 thru 0x0246df (0x2eba bytes) 000893 1788622270.769197 mapwrite 0x030b17 thru 0x037b2b (0x7015 bytes) 000896 1788622270.855553 write 0x038577 thru 0x03c4ad (0x3f37 bytes) 000897 1788622270.857465 mapwrite 0x03d7e7 thru 0x03ffff (0x2819 bytes) 000898 1788622270.982557 mapwrite 0x007d2a thru 0x014ae0 (0xcdb7 bytes) 000899 1788622271.039432 read 0x015cd4 thru 0x021ded (0xc11a bytes) 000900 1788622271.040332 read 0x028f7d thru 0x036ae9 (0xdb6d bytes) 000903 1788622271.078212 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe93e9b11 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x264b6e8d Mirror 3: 0x264b6e8d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000906 1788622272.308067 mapwrite 0x021123 thru 0x0232d7 (0x21b5 bytes) 000907 1788622272.553417 mapread 0x00965f thru 0x015350 (0xbcf2 bytes) 000909 1788622272.583686 read 0x022195 thru 0x02f2e5 (0xd151 bytes) 000911 1788622272.592184 mapwrite 0x02def9 thru 0x02ea40 (0xb48 bytes) 000912 1788622272.617940 mapread 0x0228bd thru 0x024706 (0x1e4a bytes) 000914 1788622272.618976 trunc from 0x040000 to 0x00abf1 (0x3540e bytes) 000915 1788622272.663530 read 0x009ea7 thru 0x00abf1 (0xd4b bytes) 000917 1788622272.672888 MIRROR_RESYNC 3 mirrors 000920 1788622273.478100 write 0x028bad thru 0x02cdc7 (0x421b bytes) 000921 1788622273.598460 write 0x013fd4 thru 0x016960 (0x298d bytes) 000923 1788622273.631569 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2cdc8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cdc8): Mirror 2: 0xba60c6e (stale) Mirror 3: 0xba60c6e (stale) CRC-32 checksum value for chunk [0, 0x2cdc8): Mirror 1: 0x4f02d06a Mirror 4: 0x4f02d06a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cdc8: skipped 000926 1788622275.184038 mapread 0x01bf5d thru 0x027114 (0xb1b8 bytes) 000927 1788622275.262260 write 0x034819 thru 0x03ffff (0xb7e7 bytes) 000929 1788622275.407245 MIRROR_RESYNC 4 mirrors 000932 1788622277.394282 trunc from 0x040000 to 0x00854c (0x37ab3 bytes) 000933 1788622277.501109 read 0x003c9a thru 0x0053a5 (0x170c bytes) 000935 1788622277.508495 mapread 0x00550d thru 0x00854c (0x3040 bytes) 000936 1788622277.527090 mapwrite 0x033437 thru 0x03bfe6 (0x8bb0 bytes) 000937 1788622277.605721 mapread 0x0196f8 thru 0x021657 (0x7f60 bytes) 000940 1788622277.616789 read 0x016550 thru 0x02004a (0x9afb bytes) 000943 1788622277.619198 mapread 0x011062 thru 0x01dcc6 (0xcc65 bytes) 000946 1788622277.630075 read 0x030ed3 thru 0x0380ca (0x71f8 bytes) 000948 1788622277.631305 read 0x035954 thru 0x03bfe6 (0x6693 bytes) 000949 1788622277.632378 read 0x013154 thru 0x021b8a (0xea37 bytes) 000950 1788622277.633323 mapread 0x016da8 thru 0x02496d (0xdbc6 bytes) 000954 1788622277.639975 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bfe7): Mirror 3: 0x4333b1b8 (stale) Mirror 4: 0x4333b1b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfe7): Mirror 3: 0x4333b1b8 (stale) Mirror 4: 0x4333b1b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfe7: skipped 000955 1788622277.925338 write 0x01e49a thru 0x028a00 (0xa567 bytes) 000956 1788622277.938452 MIRROR_RESYNC 3 mirrors 000957 1788622278.829434 write 0x030361 thru 0x03634f (0x5fef bytes) 000958 1788622278.897789 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bfe7): Mirror 3: 0xb1124e9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfe7): Mirror 3: 0xb1124e9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfe7: skipped 000962 1788622279.157440 mapwrite 0x03aeb3 thru 0x03ffff (0x514d bytes) 000963 1788622279.322300 MIRROR_RESYNC 2 mirrors 000964 1788622281.215439 mapread 0x02e91e thru 0x037e27 (0x950a bytes) 000966 1788622281.245314 read 0x0279ff thru 0x034edf (0xd4e1 bytes) 000969 1788622281.250695 mapread 0x02e753 thru 0x03e1e5 (0xfa93 bytes) 000971 1788622281.260985 trunc from 0x040000 to 0x00eee8 (0x31117 bytes) 000972 1788622281.331451 trunc from 0x00eee9 to 0x0075dc (0x790c bytes) 000974 1788622281.352403 mapwrite 0x01a3e9 thru 0x020196 (0x5dae bytes) 000975 1788622281.428477 trunc from 0x020197 to 0x034877 (0x146e1 bytes) 000976 1788622281.450204 MIRROR_RESYNC 2 mirrors 000978 1788622282.266196 mapread 0x013369 thru 0x01c4e2 (0x917a bytes) 000979 1788622282.286772 read 0x014e68 thru 0x02318e (0xe327 bytes) 000980 1788622282.299309 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23d:0x0]: Invalid argument (22) 000981 1788622282.437816 write 0x0190cb thru 0x019317 (0x24d bytes) 000984 1788622282.456967 write 0x0370fa thru 0x03ffff (0x8f06 bytes) 000985 1788622282.462843 MIRROR_EXTEND to 2 mirrors 000987 1788622283.668542 mapwrite 0x00f8c7 thru 0x01bcf5 (0xc42f bytes) 000988 1788622284.019390 read 0x03d418 thru 0x03ffff (0x2be8 bytes) 000989 1788622284.044368 MIRROR_RESYNC 2 mirrors 000992 1788622285.512631 read 0x0396ec thru 0x03ffff (0x6914 bytes) 000993 1788622285.554570 MIRROR_EXTEND to 3 mirrors 000994 1788622286.631176 mapread 0x021541 thru 0x023548 (0x2008 bytes) 000995 1788622286.724065 mapread 0x031000 thru 0x03a9d0 (0x99d1 bytes) 000996 1788622286.840292 read 0x02bd2d thru 0x033d78 (0x804c bytes) 000997 1788622286.861398 mapread 0x022e1a thru 0x028ed0 (0x60b7 bytes) 000999 1788622286.957793 write 0x024b8a thru 0x02699d (0x1e14 bytes) 001000 1788622287.092267 mapread 0x002b8f thru 0x006373 (0x37e5 bytes) All operations completed A-OK! PASS 70a (96s) == sanity-flr test 71: check mirror extend parent fid ==== 11:31:35 (1788622295) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 11:31:39 (1788622299) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.271042 s, 7.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245048 s, 4.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 11:31:51 (1788622311) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6920320 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7978944 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7073664 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8694624 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7856192 .. Extending file size to 8122112 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8927904 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8508032 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9062048 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9309088 .. Waiting 58163 58164 58165 58169 58170 done 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) 192.168.202.128@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:33:19 (1788622399) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6557600 .. Extending file size to 4168192 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5468320 .. Extending file size to 6132672 .. Extending file size to 8768800 .. Extending file size to 6612768 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7372864 .. Extending file size to 8500800 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8624032 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8922240 .. Waiting 59606 59607 59609 59610 59612 done 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) 192.168.202.128@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:34:37 (1788622477) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0616672 s, 34.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.286201 s, 7.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x8054305 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:34:54 (1788622494) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143386 s, 14.6 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 11:35:01 (1788622501) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x288:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x363:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x363:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x364:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x33f:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:35:08 (1788622508) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170427 s, 12.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x365:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x340:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:35:14 (1788622514) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197735 s, 10.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x367:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x367:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:35:21 (1788622521) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.303604 s, 6.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x368:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x368:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x345:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:35:54 (1788622554) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108413 s, 9.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x369:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x369:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36a:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:36:51 (1788622611) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.3462 s, 6.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:37:48 (1788622668) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg228-server mds-ost sync done. Creating new pool test_204f oleg228-server: Pool lustre.test_204f created Adding targets to pool oleg228-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg228-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.411366 s, 5.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg228-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg228-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg228-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:39:10 (1788622750) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:39:17 (1788622757) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:39:25 (1788622765) oleg228-server: Pool lustre.test_206 created Adding targets to pool oleg228-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg228-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg228-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg228-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg228-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:39:54 (1788622794) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101053 s, 10.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933864 s, 11.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf82ce297 Mirror 3: 0xf82ce297 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:40:08 (1788622808) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:40:12 (1788622812) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:40:15 (1788622815) Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.123252 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1fc0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1520000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1520000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1520000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x20b0000 Read this (4 bytes): BBBB 192.168.202.128@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg228-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:40:58 (1788622858) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0249623 s, 328 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:41:07 (1788622867) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694728 s, 15.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2ad lmm_fid: [0x200000404:0x2ad:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2ad lmm_fid: [0x200000404:0x2ad:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x376:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834605 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 11:41:09.000000000 -0400 Modify: 2026-09-05 11:41:09.000000000 -0400 Change: 2026-09-05 11:41:10.000000000 -0400 Birth: 2026-09-05 11:41:10.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:41:16 (1788622876) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:41:20 (1788622880) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.428624 s, 95.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0369722 s, 111 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 11:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2af:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 11:41 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:41:27 (1788622887) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:41:35 (1788622895) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1744 sec =========== 11:41:45 (1788622905) === sanity-flr: start cleanup 11:41:47 (1788622907) === === sanity-flr: finish cleanup 11:41:53 (1788622913) ===