-----============= acceptance-small: sanity-flr ============----- Wed Dec 3 14:44:09 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg450-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:44:22 (1764791062) === oleg450-client.virtnet: executing check_config_client /mnt/lustre oleg450-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg450-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e7906303800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e7906303800.idle_timeout=debug disable quota as required oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:44:34 (1764791074) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7545] == sanity-flr test 0a: lfs mirror create with -N option == 14:44:35 (1764791075) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:44:40 (1764791080) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:44:41 (1764791081) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:44:42 (1764791082) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:44:50 (1764791090) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:44:51 (1764791091) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 14:44:52 (1764791092) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8e7906303800.stats=clear osc.lustre-OST0001-osc-ffff8e7906303800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:44:55 (1764791095) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:44:59 (1764791099) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:45:03 (1764791103) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:45:04 (1764791104) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 14:45:08 (1764791108) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:45:11 (1764791111) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:45:12 (1764791112) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:45:15 (1764791115) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:45:19 (1764791119) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:45:22 (1764791122) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.634246 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467312 s, 22.4 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:45:27 (1764791127) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.197531 s, 95.6 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:45:31 (1764791131) Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e7906303800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e7906303800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e7906303800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e7906303800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (51s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:46:21 (1764791181) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.270822 s, 38.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e7906303800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e7906303800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (34s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:46:56 (1764791216) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e7906303800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e7906303800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e7906303800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e7906303800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (60s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:47:56 (1764791276) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.83885 s, 18.5 MB/s -rw-r--r-- 1 root root 52428800 Dec 3 14:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.832s user 0m0.000s sys 0m0.172s Stopping /mnt/lustre-ost1 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e7906303800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e7906303800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.676s user 0m0.000s sys 0m0.164s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg450-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg450-server oleg450-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e7906303800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e7906303800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.738s user 0m0.000s sys 0m0.179s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg450-server: oleg450-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg450-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e7906303800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e7906303800.ost_server_uuid in FULL state after 0 sec PASS 33b (68s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:49:04 (1764791344) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:49:09 (1764791349) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:49:14 (1764791354) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 14:49:19 (1764791359) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 14:49:28 (1764791368) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.629164 s, 20.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:49:41 (1764791381) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168129 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293779 s, 35.7 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:49:48 (1764791388) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:49:56 (1764791396) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00405922 s, 252 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00404973 s, 253 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000754272 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000699621 s, 1.5 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 14:50:05 (1764791405) create test files with size 15752840 .. e157ea90263016ca68bf065633a67c80 - 3c3cb6eea231134adca44dc5437c17a9 - a1e3e07d69b54840b177d6c9449c60db - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre PASS 37 (18s) == sanity-flr test 38: resync ============================ 14:50:23 (1764791423) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:50:35 (1764791435) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:50:41 (1764791441) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116092 s, 9.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347245 s, 30.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 14:50:49 (1764791449) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0995912 s, 21.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.230764 s, 18.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.210639 s, 10.0 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (11s) == sanity-flr test 42: lfs mirror verify ================= 14:51:00 (1764791460) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 14:51:03 (1764791463) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:51:06 (1764791466) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271386 s, 38.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 14:51:12 (1764791472) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:51:15 (1764791475) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4f:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x51:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:51:23 (1764791483) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.307496 s, 34.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:51:32 (1764791492) llite.lustre-ffff8e7906307000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.258496 s, 40.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8e79192ec000.xattr_cache=1 PASS 44e (9s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:51:41 (1764791501) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (26s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:52:07 (1764791527) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (19s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:52:26 (1764791546) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 14:52:28 (1764791548) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.12787 s, 24.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0803802 s, 39.1 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 6e0baa05ab752e147bf8870407eba373 - mirror 1 checksum: 551cf5759f33b4c80e8eb2ad0548c2f4 - mirror 2 checksum: 6e0baa05ab752e147bf8870407eba373 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Dec 3 14:52:29 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Dec 3 14:52:29 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6e0baa05ab752e147bf8870407eba373 - mirror 1 checksum: 551cf5759f33b4c80e8eb2ad0548c2f4 - mirror 2 checksum: 551cf5759f33b4c80e8eb2ad0548c2f4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Dec 3 14:52:29 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Dec 3 14:52:29 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 48 (5s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:52:34 (1764791554) Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server 14:52:37 (1764791557) shut down facet: mds1 facet_host: oleg450-server facet_failover_host: oleg450-server Failover mds1 to oleg450-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 14:52:54 (1764791574) targets are mounted 14:52:54 (1764791574) facet_failover done oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378884 s, 27.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 50A (31s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:53:05 (1764791585) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:53:07 (1764791587) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:53:09 (1764791589) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0111032 s, 1.5 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:53:15 (1764791595) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:53:16 (1764791596) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:53:18 (1764791598) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (10s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:53:28 (1764791608) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:53:48 (1764791628) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:54:06 (1764791646) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:54:24 (1764791664) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 14:54:27 (1764791667) mirror create pid 52222 mirror split pid 52223 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52222 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 52223 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 14:55:34 (1764791734) Seed set to 1736 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000006 1764791736.877530 mapwrite 0x02572f thru 0x032586 (0xce58 bytes) 000007 1764791736.999944 MIRROR_EXTEND to 2 mirrors 000009 1764791737.835272 mapread 0x02a81e thru 0x032586 (0x7d69 bytes) 000010 1764791737.858817 write 0x015d60 thru 0x01dc14 (0x7eb5 bytes) 000013 1764791737.998176 read 0x00760e thru 0x0131a0 (0xbb93 bytes) 000016 1764791738.012818 mapread 0x01b342 thru 0x02034c (0x500b bytes) truncating to largest ever: 0x1cda2 000017 1764791738.034957 trunc from 0x032587 to 0x01cda1 (0x157e5 bytes) 000019 1764791738.066859 read 0x0173e1 thru 0x01cda1 (0x59c1 bytes) 000020 1764791738.068811 write 0x03e2ba thru 0x03ffff (0x1d46 bytes) 000021 1764791738.072158 mapread 0x004ebf thru 0x01296d (0xdaaf bytes) 000022 1764791738.096386 write 0x01d71d thru 0x0210d3 (0x39b7 bytes) 000023 1764791738.097426 read 0x0120d2 thru 0x013af2 (0x1a21 bytes) 000025 1764791738.098293 write 0x032e73 thru 0x03acbd (0x7e4b bytes) 000027 1764791738.101902 mapwrite 0x03dd34 thru 0x03ffff (0x22cc bytes) 000030 1764791738.176305 write 0x02431c thru 0x0273c9 (0x30ae bytes) 000035 1764791738.183687 mapwrite 0x03a2d8 thru 0x03ffff (0x5d28 bytes) 000036 1764791738.267727 mapwrite 0x01cdd1 thru 0x027013 (0xa243 bytes) 000037 1764791738.330271 trunc from 0x040000 to 0x003f3d (0x3c0c2 bytes) 000038 1764791738.354685 read 0x0029a8 thru 0x003f3d (0x1596 bytes) 000039 1764791738.356026 mapread 0x00029a thru 0x003f3d (0x3ca4 bytes) 000040 1764791738.369951 write 0x00b110 thru 0x014759 (0x964a bytes) 000041 1764791738.379913 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000042 1764791738.589417 write 0x016926 thru 0x01f515 (0x8bf0 bytes) 000045 1764791738.593315 read 0x01b8b1 thru 0x01f515 (0x3c65 bytes) truncating to largest ever: 0x387e4 000048 1764791738.596885 trunc from 0x01f516 to 0x0387e3 (0x192ce bytes) 000050 1764791738.621619 mapread 0x010293 thru 0x01ae1a (0xab88 bytes) 000051 1764791738.637789 read 0x017e76 thru 0x020cd8 (0x8e63 bytes) 000054 1764791738.654464 mapwrite 0x00e306 thru 0x015463 (0x715e bytes) 000056 1764791738.704989 trunc from 0x0387e4 to 0x026522 (0x122c1 bytes) 000059 1764791738.721521 write 0x030bdd thru 0x031693 (0xab7 bytes) 000060 1764791738.722491 mapwrite 0x016a8f thru 0x020043 (0x95b5 bytes) 000061 1764791738.772463 mapwrite 0x01f85d thru 0x021335 (0x1ad9 bytes) 000064 1764791738.810443 read 0x01df14 thru 0x021a34 (0x3b21 bytes) 000067 1764791738.812223 write 0x010c0d thru 0x01da8d (0xce81 bytes) 000069 1764791738.814861 mapread 0x013b69 thru 0x01b9c9 (0x7e61 bytes) 000071 1764791738.823123 MIRROR_EXTEND to 2 mirrors 000072 1764791739.442667 read 0x00e104 thru 0x015f92 (0x7e8f bytes) 000073 1764791739.454012 read 0x022a8e thru 0x023489 (0x9fc bytes) 000074 1764791739.471383 trunc from 0x031694 to 0x019b72 (0x17b21 bytes) 000076 1764791739.594112 trunc from 0x019b73 to 0x011dbf (0x7db3 bytes) 000077 1764791739.629528 trunc from 0x011dc0 to 0x00fde8 (0x1fd7 bytes) 000080 1764791739.662345 trunc from 0x00fde9 to 0x01304d (0x3265 bytes) 000083 1764791739.685328 read 0x00570c thru 0x00baa5 (0x639a bytes) 000086 1764791739.706640 mapread 0x006757 thru 0x01304d (0xc8f7 bytes) 000087 1764791739.707530 mapwrite 0x0231be thru 0x0303cf (0xd212 bytes) 000088 1764791739.828383 trunc from 0x0303d0 to 0x02fd57 (0x678 bytes) 000090 1764791739.855438 mapwrite 0x03b495 thru 0x03ffff (0x4b6b bytes) 000091 1764791739.947459 trunc from 0x040000 to 0x0063b5 (0x39c4a bytes) 000092 1764791739.972616 write 0x039605 thru 0x03f75a (0x6156 bytes) 000093 1764791739.973816 mapread 0x01587d thru 0x017e47 (0x25cb bytes) 000094 1764791740.040644 read 0x03b02e thru 0x03f75a (0x472d bytes) 000096 1764791740.041627 trunc from 0x03f75b to 0x020035 (0x1f725 bytes) 000098 1764791740.065497 mapwrite 0x03b736 thru 0x03ffff (0x48ca bytes) 000099 1764791740.177279 write 0x01aab4 thru 0x029fec (0xf539 bytes) 000100 1764791740.178545 mapwrite 0x0055a0 thru 0x009b34 (0x4595 bytes) 000101 1764791740.310647 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000106 1764791740.504164 trunc from 0x040000 to 0x0311ff (0xee00 bytes) 000107 1764791740.535660 mapread 0x02d2dd thru 0x0311ff (0x3f23 bytes) 000109 1764791740.555563 mapwrite 0x03d3c2 thru 0x03ffff (0x2c3e bytes) 000110 1764791740.639404 write 0x027ddd thru 0x02bc64 (0x3e88 bytes) 000112 1764791740.647277 mapread 0x009795 thru 0x017761 (0xdfcd bytes) 000113 1764791740.664300 trunc from 0x040000 to 0x02ec40 (0x113bf bytes) 000114 1764791740.685488 trunc from 0x02ec41 to 0x024f92 (0x9cae bytes) 000115 1764791740.712315 read 0x01e254 thru 0x024f92 (0x6d3f bytes) 000116 1764791740.729361 trunc from 0x024f93 to 0x02f39c (0xa40a bytes) 000117 1764791740.749826 mapread 0x02ed2a thru 0x02f39c (0x673 bytes) 000119 1764791740.766430 read 0x010d67 thru 0x012716 (0x19b0 bytes) 000122 1764791740.767790 mapread 0x0087d6 thru 0x00f3b7 (0x6be2 bytes) 000123 1764791740.768600 trunc from 0x02f39d to 0x00b832 (0x23b6a bytes) 000124 1764791740.799134 mapread 0x00452d thru 0x00738d (0x2e61 bytes) 000126 1764791740.800334 mapread 0x001969 thru 0x0033e5 (0x1a7d bytes) 000127 1764791740.810985 trunc from 0x00b833 to 0x028146 (0x1c914 bytes) 000128 1764791740.841095 trunc from 0x028147 to 0x036066 (0xdf20 bytes) 000129 1764791740.873845 mapwrite 0x014000 thru 0x01b3bf (0x73c0 bytes) 000130 1764791741.001503 read 0x00160c thru 0x00edce (0xd7c3 bytes) 000131 1764791741.002210 read 0x015e16 thru 0x0230ef (0xd2da bytes) 000133 1764791741.003274 write 0x028f2c thru 0x02ec24 (0x5cf9 bytes) 000134 1764791741.016723 MIRROR_EXTEND to 2 mirrors 000135 1764791741.744274 write 0x035fd5 thru 0x03a870 (0x489c bytes) 000136 1764791741.834488 mapwrite 0x019efc thru 0x0276ce (0xd7d3 bytes) 000144 1764791741.953321 mapread 0x02ece0 thru 0x0303dd (0x16fe bytes) 000145 1764791741.961248 read 0x01209d thru 0x01250d (0x471 bytes) 000148 1764791741.969392 mapwrite 0x03c47d thru 0x03ffff (0x3b83 bytes) 000150 1764791742.079832 write 0x03ed45 thru 0x03f76c (0xa28 bytes) 000151 1764791742.082839 write 0x012013 thru 0x0208da (0xe8c8 bytes) 000152 1764791742.084367 mapwrite 0x03a680 thru 0x03dad1 (0x3452 bytes) 000153 1764791742.135138 MIRROR_EXTEND to 3 mirrors 000157 1764791742.798666 trunc from 0x040000 to 0x0369cb (0x9634 bytes) 000159 1764791742.938536 write 0x01ce2b thru 0x029c77 (0xce4d bytes) 000161 1764791742.985937 MIRROR_EXTEND to 4 mirrors 000164 1764791743.847635 write 0x009661 thru 0x015203 (0xbba3 bytes) 000166 1764791743.925933 read 0x028d74 thru 0x02eb84 (0x5e11 bytes) truncating to largest ever: 0x3a448 000167 1764791743.936872 trunc from 0x0369cc to 0x03a447 (0x3a7c bytes) 000168 1764791743.966595 MIRROR_EXTEND to 5 mirrors 000169 1764791744.491791 mapread 0x02680d thru 0x02986c (0x3060 bytes) 000170 1764791744.529007 mapwrite 0x02a442 thru 0x02ef1c (0x4adb bytes) 000172 1764791744.678533 mapread 0x017b65 thru 0x022ae8 (0xaf84 bytes) 000176 1764791744.693898 write 0x00d025 thru 0x012f57 (0x5f33 bytes) 000177 1764791744.702267 mapwrite 0x02e23a thru 0x032241 (0x4008 bytes) 000178 1764791744.761879 write 0x037084 thru 0x03ffff (0x8f7c bytes) 000179 1764791744.772022 mapwrite 0x0310d0 thru 0x038be2 (0x7b13 bytes) 000180 1764791744.821541 read 0x03054c thru 0x03b5e1 (0xb096 bytes) 000182 1764791744.822471 write 0x01e8a7 thru 0x020868 (0x1fc2 bytes) 000183 1764791744.823247 write 0x03df59 thru 0x03ffff (0x20a7 bytes) 000184 1764791744.823940 mapread 0x022334 thru 0x02627a (0x3f47 bytes) 000186 1764791744.825748 read 0x006298 thru 0x006b04 (0x86d bytes) 000187 1764791744.833327 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000189 1764791745.079014 mapwrite 0x00d04d thru 0x00fc37 (0x2beb bytes) 000190 1764791745.169370 mapread 0x039985 thru 0x03ffff (0x667b bytes) 000191 1764791745.176984 write 0x00e455 thru 0x01dfe7 (0xfb93 bytes) 000192 1764791745.189939 mapwrite 0x023f25 thru 0x02f04c (0xb128 bytes) truncating to largest ever: 0x3fd86 000193 1764791745.297592 trunc from 0x040000 to 0x03fd85 (0x27a bytes) 000195 1764791745.338274 mapwrite 0x0187a1 thru 0x022089 (0x98e9 bytes) 000196 1764791745.403465 mapread 0x01a3ad thru 0x01a908 (0x55c bytes) 000197 1764791745.405384 trunc from 0x03fd86 to 0x016c95 (0x290f0 bytes) 000198 1764791745.431842 trunc from 0x016c96 to 0x013a5a (0x323b bytes) 000200 1764791745.454762 MIRROR_EXTEND to 5 mirrors 000202 1764791746.166618 MIRROR_RESYNC 5 mirrors 000203 1764791747.007413 trunc from 0x013a5b to 0x02e214 (0x1a7ba bytes) 000205 1764791747.067648 mapread 0x00369c thru 0x00b00f (0x7974 bytes) 000206 1764791747.085956 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000207 1764791747.310133 MIRROR_EXTEND to 5 mirrors 000209 1764791747.947939 write 0x004b2f thru 0x00b91f (0x6df1 bytes) 000210 1764791748.044252 mapwrite 0x01b9f8 thru 0x01bee9 (0x4f2 bytes) 000212 1764791748.130704 MIRROR_EXTEND to 6 mirrors 000213 1764791748.621865 MIRROR_RESYNC 6 mirrors 000214 1764791749.590682 write 0x00e4ae thru 0x01d8a3 (0xf3f6 bytes) 000215 1764791749.688599 write 0x022bfe thru 0x02806a (0x546d bytes) 000219 1764791749.697884 trunc from 0x02e215 to 0x03a051 (0xbe3d bytes) 000220 1764791749.740291 mapread 0x004bdf thru 0x012456 (0xd878 bytes) 000221 1764791749.748602 write 0x0272e3 thru 0x027a2c (0x74a bytes) 000222 1764791749.749132 mapread 0x00e220 thru 0x01244f (0x4230 bytes) 000225 1764791749.750681 read 0x031373 thru 0x035250 (0x3ede bytes) 000227 1764791749.761586 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000228 1764791749.954306 mapread 0x02a12f thru 0x02de2c (0x3cfe bytes) 000231 1764791749.970307 mapwrite 0x01c533 thru 0x0279b1 (0xb47f bytes) 000232 1764791750.023124 mapwrite 0x020de9 thru 0x02ec04 (0xde1c bytes) 000233 1764791750.060005 read 0x001df6 thru 0x00c0a2 (0xa2ad bytes) 000235 1764791750.072468 write 0x0093fd thru 0x016812 (0xd416 bytes) 000236 1764791750.073874 mapread 0x036933 thru 0x038c66 (0x2334 bytes) 000237 1764791750.091702 trunc from 0x03a052 to 0x009a93 (0x305be bytes) 000238 1764791750.108715 mapread 0x005d9e thru 0x009a93 (0x3cf6 bytes) 000240 1764791750.114743 MIRROR_RESYNC 5 mirrors 000241 1764791750.761550 write 0x03fc6a thru 0x03ffff (0x396 bytes) 000242 1764791750.800166 mapread 0x02a2cf thru 0x02aa1f (0x751 bytes) 000243 1764791750.802999 write 0x03c8e8 thru 0x03ffff (0x3718 bytes) 000244 1764791750.809777 trunc from 0x040000 to 0x02507d (0x1af82 bytes) 000248 1764791750.834377 mapread 0x011af1 thru 0x0211cb (0xf6db bytes) 000250 1764791750.863009 MIRROR_EXTEND to 6 mirrors 000253 1764791751.581685 mapread 0x01ccec thru 0x01dfe2 (0x12f7 bytes) 000255 1764791751.612157 write 0x03f40e thru 0x03ffff (0xbf2 bytes) 000257 1764791751.700580 mapwrite 0x01da84 thru 0x0231e5 (0x5762 bytes) 000263 1764791751.801295 trunc from 0x040000 to 0x0061a5 (0x39e5a bytes) 000266 1764791751.832945 mapwrite 0x0160a4 thru 0x022950 (0xc8ad bytes) 000269 1764791751.936663 MIRROR_RESYNC 6 mirrors 000270 1764791752.715836 trunc from 0x022951 to 0x005966 (0x1cfea bytes) 000272 1764791752.791539 read 0x004ee1 thru 0x005966 (0xa86 bytes) 000274 1764791752.796929 trunc from 0x005967 to 0x038a86 (0x33120 bytes) 000275 1764791752.815106 write 0x0014e4 thru 0x008f23 (0x7a40 bytes) 000276 1764791752.825214 mapwrite 0x012397 thru 0x01b7fd (0x9467 bytes) 000278 1764791752.905179 mapread 0x02abc0 thru 0x03322c (0x866d bytes) 000279 1764791752.929597 mapwrite 0x03c0da thru 0x03ffff (0x3f26 bytes) 000284 1764791753.033839 mapread 0x036bf5 thru 0x03ffff (0x940b bytes) 000285 1764791753.035753 mapwrite 0x006b24 thru 0x00d7e5 (0x6cc2 bytes) 000286 1764791753.103754 mapwrite 0x00b9f7 thru 0x014dc2 (0x93cc bytes) 000288 1764791753.160315 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000290 1764791753.414097 read 0x0349ae thru 0x039a9c (0x50ef bytes) 000291 1764791753.422302 mapwrite 0x026c09 thru 0x0313f0 (0xa7e8 bytes) 000294 1764791753.500772 mapwrite 0x038b44 thru 0x03aaea (0x1fa7 bytes) 000295 1764791753.579008 write 0x02576a thru 0x02bfd1 (0x6868 bytes) 000296 1764791753.580098 read 0x03da6e thru 0x03ffff (0x2592 bytes) 000299 1764791753.582547 mapread 0x027bb3 thru 0x03692b (0xed79 bytes) 000301 1764791753.584876 write 0x000464 thru 0x005243 (0x4de0 bytes) 000302 1764791753.594814 trunc from 0x040000 to 0x038dc5 (0x723a bytes) 000303 1764791753.616619 read 0x01d13a thru 0x0232c8 (0x618f bytes) 000304 1764791753.620881 MIRROR_EXTEND to 6 mirrors 000307 1764791754.160353 MIRROR_RESYNC 6 mirrors 000309 1764791754.999020 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000310 1764791755.311198 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000312 1764791755.612366 MIRROR_EXTEND to 5 mirrors 000314 1764791756.401811 MIRROR_RESYNC 5 mirrors 000315 1764791756.786809 mapread 0x00bb4f thru 0x0195a3 (0xda55 bytes) 000316 1764791756.859493 trunc from 0x038dc6 to 0x029fff (0xedc6 bytes) 000317 1764791757.150703 mapread 0x007842 thru 0x00e7a9 (0x6f68 bytes) 000318 1764791757.185536 write 0x006290 thru 0x00f15d (0x8ece bytes) 000319 1764791757.204178 read 0x01e69e thru 0x0205ba (0x1f1d bytes) 000321 1764791757.213901 read 0x00b89c thru 0x010eef (0x5654 bytes) 000323 1764791757.214927 mapwrite 0x023924 thru 0x02961d (0x5cfa bytes) 000324 1764791757.380484 write 0x0375d3 thru 0x03ffff (0x8a2d bytes) 000325 1764791757.399870 MIRROR_RESYNC 5 mirrors 000326 1764791758.625432 write 0x00799c thru 0x010573 (0x8bd8 bytes) 000327 1764791758.694151 read 0x01e44b thru 0x01ec66 (0x81c bytes) 000328 1764791758.698647 read 0x01d013 thru 0x02c52b (0xf519 bytes) 000330 1764791758.714994 write 0x009c37 thru 0x019183 (0xf54d bytes) 000333 1764791758.727660 mapwrite 0x033ccc thru 0x03de33 (0xa168 bytes) 000334 1764791758.836437 read 0x0367e2 thru 0x03d9f8 (0x7217 bytes) 000335 1764791758.836967 trunc from 0x040000 to 0x02bf02 (0x140fd bytes) 000336 1764791758.933947 mapwrite 0x03b9d7 thru 0x03ffff (0x4629 bytes) 000337 1764791759.168470 trunc from 0x040000 to 0x010858 (0x2f7a7 bytes) 000339 1764791759.205792 mapread 0x00c067 thru 0x010858 (0x47f2 bytes) 000341 1764791759.217357 trunc from 0x010859 to 0x012897 (0x203f bytes) 000343 1764791759.275661 mapwrite 0x03ab83 thru 0x03fbab (0x5029 bytes) 000344 1764791759.482145 MIRROR_RESYNC 5 mirrors 000346 1764791760.892994 MIRROR_RESYNC 5 mirrors 000348 1764791761.423678 write 0x017623 thru 0x023d82 (0xc760 bytes) 000349 1764791761.605173 mapwrite 0x026468 thru 0x034238 (0xddd1 bytes) 000350 1764791761.819581 trunc from 0x03fbac to 0x01b9a1 (0x2420a bytes) 000353 1764791761.879146 read 0x003389 thru 0x004577 (0x11ef bytes) 000354 1764791761.898741 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000355 1764791762.359802 write 0x0137a3 thru 0x01d320 (0x9b7e bytes) 000358 1764791762.379463 mapread 0x01caa9 thru 0x01d320 (0x878 bytes) 000359 1764791762.381819 write 0x0291da thru 0x02e4d4 (0x52fb bytes) 000360 1764791762.384395 mapwrite 0x004326 thru 0x01126f (0xcf4a bytes) 000362 1764791762.537091 MIRROR_EXTEND to 5 mirrors 000365 1764791763.504491 mapwrite 0x019a80 thru 0x01ba9c (0x201d bytes) 000366 1764791763.700754 mapread 0x019529 thru 0x026868 (0xd340 bytes) 000368 1764791763.730009 mapwrite 0x037686 thru 0x03ea7a (0x73f5 bytes) 000370 1764791763.885056 read 0x010a9b thru 0x017867 (0x6dcd bytes) 000372 1764791763.908731 read 0x02199f thru 0x021fb3 (0x615 bytes) 000373 1764791763.909309 trunc from 0x03ea7b to 0x006f97 (0x37ae3 bytes) 000374 1764791763.960402 write 0x03e146 thru 0x03ffff (0x1eba bytes) 000375 1764791763.961351 write 0x02258e thru 0x026fa8 (0x4a1b bytes) 000376 1764791763.983438 read 0x03fbca thru 0x03ffff (0x436 bytes) 000378 1764791764.001931 MIRROR_EXTEND to 6 mirrors 000379 1764791765.184008 write 0x00d2d1 thru 0x00dc24 (0x954 bytes) 000380 1764791765.349929 write 0x01a731 thru 0x0256f9 (0xafc9 bytes) 000381 1764791765.381972 mapwrite 0x00e175 thru 0x00ed4f (0xbdb bytes) 000382 1764791765.543685 trunc from 0x040000 to 0x00ac01 (0x353fe bytes) 000383 1764791765.578524 mapwrite 0x018413 thru 0x02535a (0xcf48 bytes) 000385 1764791765.791706 mapwrite 0x024233 thru 0x02aac1 (0x688f bytes) 000387 1764791766.010268 mapwrite 0x00d0bd thru 0x0131c2 (0x6106 bytes) 000388 1764791766.206248 write 0x036ddb thru 0x03ffff (0x9225 bytes) 000390 1764791766.212533 mapread 0x011d37 thru 0x01d402 (0xb6cc bytes) 000391 1764791766.224060 mapwrite 0x02ed0f thru 0x02f614 (0x906 bytes) 000393 1764791766.465337 trunc from 0x040000 to 0x01b0cb (0x24f34 bytes) 000394 1764791766.531294 mapwrite 0x02811c thru 0x03638b (0xe270 bytes) 000395 1764791766.837600 trunc from 0x03638c to 0x03db8b (0x7800 bytes) 000396 1764791766.909480 mapwrite 0x03ae0c thru 0x03ffff (0x51f4 bytes) 000398 1764791767.129511 mapwrite 0x035454 thru 0x039263 (0x3e10 bytes) 000400 1764791767.298964 mapread 0x02ce71 thru 0x02d1bf (0x34f bytes) 000403 1764791767.318456 mapread 0x00d2ab thru 0x00fc3a (0x2990 bytes) 000404 1764791767.321150 mapread 0x001115 thru 0x006e4a (0x5d36 bytes) 000406 1764791767.369078 write 0x02cae0 thru 0x037e63 (0xb384 bytes) 000408 1764791767.376101 write 0x00be6c thru 0x00ea6a (0x2bff bytes) 000409 1764791767.376910 read 0x03243e thru 0x03ffff (0xdbc2 bytes) 000411 1764791767.381344 trunc from 0x040000 to 0x00328f (0x3cd70 bytes) 000413 1764791767.416810 read 0x000fa9 thru 0x00328f (0x22e7 bytes) 000414 1764791767.430132 mapwrite 0x0168af thru 0x01ff52 (0x96a4 bytes) 000418 1764791767.637916 mapread 0x00b455 thru 0x011f79 (0x6b25 bytes) 000419 1764791767.645581 mapread 0x00a08b thru 0x0188d2 (0xe848 bytes) 000421 1764791767.659994 mapread 0x00c232 thru 0x015fa9 (0x9d78 bytes) 000423 1764791767.661585 write 0x0174e7 thru 0x01ec82 (0x779c bytes) 000425 1764791767.665673 mapread 0x01f452 thru 0x01ff52 (0xb01 bytes) 000426 1764791767.669955 mapwrite 0x022b88 thru 0x031a88 (0xef01 bytes) 000428 1764791767.904602 trunc from 0x031a89 to 0x013b65 (0x1df23 bytes) 000430 1764791767.955677 mapread 0x006304 thru 0x00e83f (0x853c bytes) 000432 1764791767.962378 mapread 0x011932 thru 0x0119b4 (0x83 bytes) 000433 1764791767.988592 MIRROR_RESYNC 6 mirrors 000435 1764791770.133324 mapwrite 0x00a69f thru 0x0104c5 (0x5e27 bytes) 000436 1764791770.461872 read 0x0060e9 thru 0x009154 (0x306c bytes) 000437 1764791770.463652 write 0x0166bc thru 0x025f54 (0xf899 bytes) 000438 1764791770.469750 trunc from 0x025f55 to 0x00ea0d (0x17547 bytes) 000439 1764791770.500881 mapwrite 0x0143ca thru 0x023982 (0xf5b9 bytes) 000442 1764791770.607784 MIRROR_RESYNC 6 mirrors 000444 1764791772.661494 write 0x023fc1 thru 0x026e6e (0x2eae bytes) 000452 1764791772.814522 mapread 0x014996 thru 0x01acaa (0x6315 bytes) 000453 1764791772.859552 mapwrite 0x02eae2 thru 0x0370b4 (0x85d3 bytes) 000454 1764791773.140200 write 0x028d87 thru 0x031e48 (0x90c2 bytes) 000455 1764791773.147519 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000457 1764791773.630885 trunc from 0x0370b5 to 0x012083 (0x25031 bytes) 000458 1764791773.687080 mapread 0x00d6a8 thru 0x012083 (0x49dc bytes) 000459 1764791773.771588 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000462 1764791774.292825 mapwrite 0x000e08 thru 0x00c00b (0xb204 bytes) 000465 1764791774.477732 MIRROR_RESYNC 4 mirrors 000466 1764791775.259693 write 0x0024e0 thru 0x0095fd (0x711e bytes) 000467 1764791775.365402 write 0x036fb4 thru 0x038eb9 (0x1f06 bytes) 000468 1764791775.371376 trunc from 0x038eba to 0x037e85 (0x1034 bytes) 000469 1764791775.471319 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000470 1764791775.731637 mapwrite 0x037ea4 thru 0x03ffff (0x815c bytes) 000471 1764791775.857146 write 0x0012c1 thru 0x00553c (0x427c bytes) 000472 1764791775.870100 mapwrite 0x0329cb thru 0x03f290 (0xc8c6 bytes) 000473 1764791775.969871 mapwrite 0x02fe86 thru 0x038c34 (0x8daf bytes) 000474 1764791776.083966 mapwrite 0x0345d9 thru 0x03ffff (0xba27 bytes) 000475 1764791776.193568 write 0x023022 thru 0x02b2bd (0x829c bytes) 000477 1764791776.209617 write 0x029200 thru 0x02e215 (0x5016 bytes) 000480 1764791776.213950 read 0x00b078 thru 0x016f80 (0xbf09 bytes) 000482 1764791776.227726 mapwrite 0x01dfb0 thru 0x02d614 (0xf665 bytes) 000484 1764791776.336287 mapwrite 0x031615 thru 0x03ffff (0xe9eb bytes) 000486 1764791776.412747 mapread 0x006080 thru 0x014a8d (0xea0e bytes) 000488 1764791776.414093 read 0x02a638 thru 0x03116a (0x6b33 bytes) 000491 1764791776.416890 trunc from 0x040000 to 0x0373fd (0x8c02 bytes) 000492 1764791776.467432 MIRROR_RESYNC 3 mirrors 000493 1764791777.485490 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000496 1764791777.954550 read 0x01e135 thru 0x02779a (0x9666 bytes) 000500 1764791777.968914 mapread 0x01c194 thru 0x01fd98 (0x3c05 bytes) 000501 1764791777.982526 read 0x031f73 thru 0x0373fd (0x548b bytes) 000506 1764791777.994613 mapread 0x03578e thru 0x0373fd (0x1c70 bytes) 000507 1764791777.995496 write 0x025f7d thru 0x0292a4 (0x3328 bytes) 000508 1764791778.087018 mapwrite 0x01eb35 thru 0x01eed0 (0x39c bytes) 000509 1764791778.132765 write 0x03aa5a thru 0x03ffff (0x55a6 bytes) 000510 1764791778.133861 trunc from 0x040000 to 0x02cd19 (0x132e6 bytes) 000513 1764791778.158850 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000514 1764791778.276468 read 0x00f59a thru 0x01a846 (0xb2ad bytes) 000516 1764791778.309336 MIRROR_EXTEND to 2 mirrors 000518 1764791778.819256 mapread 0x010fa4 thru 0x01498e (0x39eb bytes) 000519 1764791778.838825 mapread 0x019792 thru 0x028de0 (0xf64f bytes) 000520 1764791778.875726 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000522 1764791779.101222 read 0x025aab thru 0x02cd19 (0x726f bytes) 000523 1764791779.128635 mapwrite 0x00ab9d thru 0x01034b (0x57af bytes) 000524 1764791779.278533 read 0x0149a5 thru 0x01c99f (0x7ffb bytes) 000527 1764791779.301917 mapread 0x00395f thru 0x0077b3 (0x3e55 bytes) 000528 1764791779.303743 mapread 0x018726 thru 0x024cb5 (0xc590 bytes) 000529 1764791779.306542 mapread 0x00b58d thru 0x0117e2 (0x6256 bytes) 000530 1764791779.310006 write 0x017b4b thru 0x023e37 (0xc2ed bytes) 000532 1764791779.317191 trunc from 0x02cd1a to 0x00ac3f (0x220da bytes) 000536 1764791779.419753 read 0x005d62 thru 0x00ac3f (0x4ede bytes) 000537 1764791779.420368 write 0x00ba9c thru 0x011864 (0x5dc9 bytes) 000539 1764791779.430916 mapread 0x0003e5 thru 0x00469d (0x42b9 bytes) 000543 1764791779.436615 trunc from 0x011865 to 0x0312ab (0x1fa47 bytes) 000544 1764791779.459283 read 0x0011ca thru 0x001b0f (0x946 bytes) 000545 1764791779.459805 read 0x02b311 thru 0x0312ab (0x5f9b bytes) 000546 1764791779.486137 MIRROR_EXTEND to 2 mirrors 000548 1764791780.372536 read 0x017af5 thru 0x024d64 (0xd270 bytes) 000550 1764791780.391738 mapread 0x00c5ab thru 0x018b28 (0xc57e bytes) 000551 1764791780.427079 mapread 0x030577 thru 0x0312ab (0xd35 bytes) 000554 1764791780.456946 mapread 0x010b8b thru 0x01cfa4 (0xc41a bytes) 000555 1764791780.472811 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000559 1764791780.722423 write 0x01e462 thru 0x020248 (0x1de7 bytes) 000562 1764791780.761763 mapread 0x010db8 thru 0x015ae3 (0x4d2c bytes) 000563 1764791780.778188 trunc from 0x0312ac to 0x013f11 (0x1d39a bytes) 000564 1764791780.807026 trunc from 0x013f12 to 0x03fa31 (0x2bb20 bytes) 000567 1764791780.848420 read 0x03937e thru 0x03fa31 (0x66b4 bytes) 000571 1764791780.865380 mapwrite 0x03672b thru 0x03ffff (0x98d5 bytes) 000573 1764791780.984546 mapwrite 0x02936f thru 0x02fe80 (0x6b12 bytes) 000574 1764791781.089701 mapwrite 0x03d135 thru 0x03ffff (0x2ecb bytes) 000575 1764791781.122248 mapread 0x03e614 thru 0x03ffff (0x19ec bytes) 000576 1764791781.123014 read 0x03b594 thru 0x03ffff (0x4a6c bytes) 000577 1764791781.124836 write 0x00e586 thru 0x01627a (0x7cf5 bytes) 000579 1764791781.133366 mapread 0x00f72a thru 0x019a1b (0xa2f2 bytes) 000580 1764791781.141455 mapwrite 0x016af7 thru 0x0196f0 (0x2bfa bytes) 000581 1764791781.222607 MIRROR_EXTEND to 2 mirrors 000582 1764791782.193431 trunc from 0x040000 to 0x01ff66 (0x20099 bytes) 000584 1764791782.386006 mapwrite 0x02b6d3 thru 0x02d8bc (0x21ea bytes) 000585 1764791782.555120 MIRROR_EXTEND to 3 mirrors 000586 1764791783.206584 mapread 0x029e7b thru 0x02d8bc (0x3a42 bytes) 000587 1764791783.275714 mapwrite 0x02553e thru 0x02d22f (0x7cf2 bytes) 000589 1764791783.574919 write 0x01dac8 thru 0x02186b (0x3da4 bytes) 000591 1764791783.591221 mapread 0x029e8b thru 0x02d8bc (0x3a32 bytes) 000592 1764791783.592085 write 0x0321a3 thru 0x03ed67 (0xcbc5 bytes) 000593 1764791783.606852 MIRROR_EXTEND to 4 mirrors 000594 1764791784.690666 trunc from 0x03ed68 to 0x02bf63 (0x12e04 bytes) 000599 1764791784.785156 write 0x0075d0 thru 0x00eba5 (0x75d6 bytes) 000600 1764791784.793899 trunc from 0x02bf64 to 0x0021e2 (0x29d81 bytes) 000602 1764791784.821195 write 0x004aea thru 0x00e9a1 (0x9eb8 bytes) 000604 1764791784.831199 mapread 0x001892 thru 0x00e9a1 (0xd110 bytes) 000606 1764791784.842369 write 0x000291 thru 0x00725b (0x6fcb bytes) 000607 1764791784.853465 trunc from 0x00e9a2 to 0x019833 (0xae92 bytes) 000609 1764791784.898774 trunc from 0x019834 to 0x031ea0 (0x1866d bytes) 000610 1764791784.926897 read 0x02735c thru 0x02aa73 (0x3718 bytes) 000612 1764791784.942404 mapread 0x00b5b2 thru 0x01a0eb (0xeb3a bytes) 000613 1764791784.972844 mapread 0x028987 thru 0x0317f4 (0x8e6e bytes) 000614 1764791785.001372 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000617 1764791785.244832 read 0x02c8e0 thru 0x031ea0 (0x55c1 bytes) 000619 1764791785.255157 read 0x015295 thru 0x01d692 (0x83fe bytes) 000621 1764791785.274574 mapwrite 0x02527d thru 0x027d68 (0x2aec bytes) 000624 1764791785.334961 write 0x005705 thru 0x013656 (0xdf52 bytes) 000625 1764791785.336959 mapwrite 0x013490 thru 0x0164ab (0x301c bytes) 000626 1764791785.397850 write 0x00539f thru 0x014df4 (0xfa56 bytes) 000627 1764791785.399025 mapwrite 0x022700 thru 0x0267c4 (0x40c5 bytes) 000628 1764791785.470965 MIRROR_EXTEND to 4 mirrors 000631 1764791786.338286 MIRROR_RESYNC 4 mirrors 000632 1764791787.106884 trunc from 0x031ea1 to 0x03ee3a (0xcf9a bytes) 000633 1764791787.197309 MIRROR_RESYNC 4 mirrors 000634 1764791788.320391 mapread 0x03d184 thru 0x03ee3a (0x1cb7 bytes) 000635 1764791788.344864 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000636 1764791788.665166 MIRROR_RESYNC 3 mirrors 000637 1764791788.942452 trunc from 0x03ee3b to 0x0257a6 (0x19694 bytes) 000638 1764791789.088212 read 0x01c223 thru 0x0257a6 (0x9584 bytes) 000639 1764791789.099435 mapread 0x010753 thru 0x0127bb (0x2069 bytes) 000643 1764791789.130435 trunc from 0x0257a7 to 0x0247df (0xfc7 bytes) 000645 1764791789.164640 MIRROR_EXTEND to 4 mirrors 000648 1764791789.828398 mapread 0x01adc5 thru 0x01ff96 (0x51d2 bytes) 000649 1764791789.872580 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000651 1764791790.179470 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000653 1764791790.593322 trunc from 0x0247e0 to 0x026d08 (0x2529 bytes) 000654 1764791790.680901 mapread 0x021bd9 thru 0x026d08 (0x5130 bytes) 000655 1764791790.706991 read 0x0119d7 thru 0x01c133 (0xa75d bytes) 000657 1764791790.777218 MIRROR_EXTEND to 3 mirrors 000658 1764791791.624114 read 0x00391e thru 0x003dd8 (0x4bb bytes) 000659 1764791791.639805 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000661 1764791791.914966 mapread 0x01bebf thru 0x01c3d2 (0x514 bytes) 000663 1764791791.928526 write 0x008c09 thru 0x017db6 (0xf1ae bytes) 000664 1764791792.005956 write 0x03c878 thru 0x03e956 (0x20df bytes) 000665 1764791792.011227 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000667 1764791792.130887 mapread 0x0230a6 thru 0x02c9c1 (0x991c bytes) 000669 1764791792.149744 mapwrite 0x01864f thru 0x020f39 (0x88eb bytes) 000671 1764791792.210182 mapwrite 0x03a5c6 thru 0x03ffff (0x5a3a bytes) 000672 1764791792.279023 trunc from 0x040000 to 0x027d99 (0x18266 bytes) 000674 1764791792.294937 mapwrite 0x03d384 thru 0x03ffff (0x2c7c bytes) 000676 1764791792.384301 MIRROR_EXTEND to 2 mirrors 000683 1764791793.000653 mapwrite 0x013f84 thru 0x01e94d (0xa9ca bytes) 000684 1764791793.236743 mapwrite 0x016165 thru 0x01d77f (0x761b bytes) 000688 1764791793.341572 read 0x0014c7 thru 0x00d38f (0xbec9 bytes) 000689 1764791793.356522 write 0x03132a thru 0x036aa6 (0x577d bytes) 000690 1764791793.390707 write 0x028f7f thru 0x031465 (0x84e7 bytes) 000692 1764791793.410541 trunc from 0x040000 to 0x017275 (0x28d8a bytes) 000693 1764791793.522860 mapread 0x00482e thru 0x007828 (0x2ffb bytes) 000694 1764791793.533334 mapwrite 0x00a0e2 thru 0x00bd27 (0x1c46 bytes) 000696 1764791793.650751 mapwrite 0x03ad70 thru 0x03f19f (0x4430 bytes) 000699 1764791793.820133 write 0x03529f thru 0x03af61 (0x5cc3 bytes) 000700 1764791793.821108 mapwrite 0x0163d1 thru 0x02627b (0xfeab bytes) 000705 1764791793.980606 read 0x03416e thru 0x03c74e (0x85e1 bytes) 000706 1764791793.984615 mapread 0x01db4a thru 0x02b771 (0xdc28 bytes) 000708 1764791794.004360 trunc from 0x03f1a0 to 0x02aeb6 (0x142e9 bytes) 000710 1764791794.061404 MIRROR_RESYNC 2 mirrors 000713 1764791794.613232 write 0x007efb thru 0x01670e (0xe814 bytes) 000714 1764791794.690713 mapwrite 0x02ee12 thru 0x030aaa (0x1c99 bytes) 000717 1764791794.780894 read 0x0077dc thru 0x00a2db (0x2b00 bytes) 000718 1764791794.781458 read 0x00887e thru 0x010fa5 (0x8728 bytes) 000721 1764791794.782812 write 0x02b6ae thru 0x02c83e (0x1191 bytes) 000722 1764791794.783659 trunc from 0x030aab to 0x00e175 (0x22935 bytes) 000723 1764791794.806967 read 0x0075f6 thru 0x00d3fc (0x5e07 bytes) 000724 1764791794.807579 mapread 0x00dad3 thru 0x00e175 (0x6a3 bytes) 000725 1764791794.808367 mapread 0x00b766 thru 0x00e175 (0x2a10 bytes) 000726 1764791794.809098 mapread 0x00b905 thru 0x00e175 (0x2871 bytes) 000727 1764791794.816005 write 0x0240e4 thru 0x02dcc6 (0x9be3 bytes) 000728 1764791794.818284 trunc from 0x02dcc7 to 0x010488 (0x1d83e bytes) 000729 1764791794.838902 write 0x0143ad thru 0x01b2ef (0x6f43 bytes) 000730 1764791794.840451 mapread 0x01a421 thru 0x01b2ef (0xecf bytes) 000731 1764791794.841228 trunc from 0x01b2f0 to 0x028079 (0xcd8a bytes) 000732 1764791794.863365 mapread 0x0236ab thru 0x028079 (0x49cf bytes) 000734 1764791794.872489 read 0x026d34 thru 0x028079 (0x1346 bytes) 000736 1764791794.873480 mapread 0x01a677 thru 0x01e3a2 (0x3d2c bytes) 000739 1764791794.884228 MIRROR_EXTEND to 3 mirrors 000740 1764791795.357857 mapwrite 0x0091c5 thru 0x01646f (0xd2ab bytes) 000741 1764791795.576490 MIRROR_RESYNC 3 mirrors 000743 1764791796.542102 write 0x0121b0 thru 0x017537 (0x5388 bytes) 000746 1764791796.609750 trunc from 0x02807a to 0x02f73f (0x76c6 bytes) 000751 1764791796.634812 read 0x02739f thru 0x029179 (0x1ddb bytes) 000752 1764791796.652684 mapread 0x0127ab thru 0x018868 (0x60be bytes) 000759 1764791796.677408 mapwrite 0x00d972 thru 0x01bd22 (0xe3b1 bytes) 000760 1764791796.828739 write 0x039d95 thru 0x03a0aa (0x316 bytes) 000761 1764791796.829637 mapwrite 0x00e962 thru 0x01d7be (0xee5d bytes) 000762 1764791796.898257 write 0x0147e1 thru 0x022b0c (0xe32c bytes) 000763 1764791796.903355 write 0x01f226 thru 0x023be2 (0x49bd bytes) 000764 1764791796.909900 mapwrite 0x022ea6 thru 0x030be3 (0xdd3e bytes) 000765 1764791796.959327 read 0x01468b thru 0x01c34b (0x7cc1 bytes) 000766 1764791796.959924 write 0x00deed thru 0x014e66 (0x6f7a bytes) 000769 1764791796.962503 mapwrite 0x0346d4 thru 0x034def (0x71c bytes) 000771 1764791797.015135 mapwrite 0x031d4d thru 0x03d9fe (0xbcb2 bytes) 000772 1764791797.094989 trunc from 0x03d9ff to 0x002dfb (0x3ac03 bytes) 000773 1764791797.125580 write 0x01de59 thru 0x02bda1 (0xdf49 bytes) 000774 1764791797.129766 mapwrite 0x02e14d thru 0x02e569 (0x41d bytes) 000775 1764791797.233869 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000776 1764791797.502820 read 0x01b5a2 thru 0x021716 (0x6175 bytes) 000777 1764791797.517473 mapread 0x025f6f thru 0x02e569 (0x85fb bytes) 000778 1764791797.542309 trunc from 0x02e56a to 0x0209fd (0xdb6c bytes) 000781 1764791797.569157 trunc from 0x0209fe to 0x02b0e3 (0xa6e6 bytes) 000784 1764791797.608922 read 0x023c04 thru 0x028dbd (0x51ba bytes) 000785 1764791797.623117 read 0x00828e thru 0x00ba21 (0x3794 bytes) 000786 1764791797.624886 mapread 0x01b961 thru 0x02b0e3 (0xf783 bytes) 000790 1764791797.629514 write 0x03d9d0 thru 0x03ffff (0x2630 bytes) 000792 1764791797.630867 trunc from 0x040000 to 0x037d4a (0x82b5 bytes) 000793 1764791797.660794 mapread 0x0370ae thru 0x037d4a (0xc9d bytes) 000796 1764791797.668538 mapwrite 0x021091 thru 0x02ee2d (0xdd9d bytes) 000797 1764791797.747213 trunc from 0x037d4b to 0x02e4af (0x989b bytes) 000798 1764791797.763132 read 0x026e69 thru 0x02e4af (0x7647 bytes) 000799 1764791797.769927 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000802 1764791797.966090 mapwrite 0x0036e4 thru 0x00ef0e (0xb82b bytes) 000804 1764791798.098261 write 0x0105b1 thru 0x01d703 (0xd153 bytes) 000806 1764791798.115406 mapread 0x01f857 thru 0x02d588 (0xdd32 bytes) 000808 1764791798.139859 mapwrite 0x03ae45 thru 0x03ae68 (0x24 bytes) 000810 1764791798.326738 trunc from 0x03ae69 to 0x00557f (0x358e9 bytes) 000811 1764791798.354958 mapread 0x0005ba thru 0x00557f (0x4fc6 bytes) 000813 1764791798.363781 write 0x01d484 thru 0x01fc6e (0x27eb bytes) 000814 1764791798.364905 mapread 0x0009de thru 0x006e94 (0x64b7 bytes) 000815 1764791798.383196 write 0x00800e thru 0x00c240 (0x4233 bytes) 000817 1764791798.384561 trunc from 0x01fc6f to 0x03e402 (0x1e794 bytes) 000818 1764791798.403183 mapread 0x012c82 thru 0x01c15f (0x94de bytes) 000819 1764791798.436767 read 0x01817f thru 0x023a19 (0xb89b bytes) 000820 1764791798.437374 read 0x03b2fa thru 0x03e402 (0x3109 bytes) 000822 1764791798.460285 write 0x008a8a thru 0x00f6d8 (0x6c4f bytes) 000825 1764791798.462474 mapread 0x001a30 thru 0x00ab41 (0x9112 bytes) 000826 1764791798.464255 mapread 0x00a793 thru 0x0185cd (0xde3b bytes) 000828 1764791798.466328 mapread 0x014c64 thru 0x01fc18 (0xafb5 bytes) 000830 1764791798.468273 write 0x008381 thru 0x0090cd (0xd4d bytes) 000831 1764791798.469028 trunc from 0x03e403 to 0x03bf09 (0x24f9 bytes) 000833 1764791798.506306 read 0x012a48 thru 0x0134f8 (0xab1 bytes) 000835 1764791798.510801 write 0x023e17 thru 0x0310aa (0xd294 bytes) 000836 1764791798.513235 mapwrite 0x024fc6 thru 0x034b4e (0xfb89 bytes) 000837 1764791798.639585 write 0x0298ae thru 0x02dafb (0x424e bytes) 000838 1764791798.643773 read 0x025b87 thru 0x03595f (0xfdd9 bytes) 000839 1764791798.649691 mapwrite 0x035109 thru 0x03ffff (0xaef7 bytes) 000840 1764791798.814956 write 0x019cae thru 0x022c59 (0x8fac bytes) 000841 1764791798.816136 mapwrite 0x015303 thru 0x024d19 (0xfa17 bytes) 000847 1764791798.931461 write 0x00c0ad thru 0x01826c (0xc1c0 bytes) 000850 1764791798.934864 write 0x038f2e thru 0x03ffff (0x70d2 bytes) 000852 1764791798.937618 mapwrite 0x00bc43 thru 0x00e23c (0x25fa bytes) 000853 1764791799.035668 mapwrite 0x00e1dc thru 0x01db77 (0xf99c bytes) 000855 1764791799.127650 MIRROR_EXTEND to 2 mirrors 000857 1764791800.307298 write 0x02ef76 thru 0x03036c (0x13f7 bytes) 000858 1764791800.459711 mapwrite 0x003ea2 thru 0x0136cd (0xf82c bytes) 000860 1764791800.635813 mapread 0x01c227 thru 0x0204e8 (0x42c2 bytes) 000861 1764791800.649825 mapread 0x0115d9 thru 0x01635f (0x4d87 bytes) 000862 1764791800.652835 mapread 0x00653a thru 0x01174e (0xb215 bytes) 000866 1764791800.658238 mapwrite 0x00a9e7 thru 0x0116ad (0x6cc7 bytes) 000867 1764791800.757876 trunc from 0x040000 to 0x0284a9 (0x17b56 bytes) 000869 1764791800.838425 trunc from 0x0284aa to 0x029053 (0xbaa bytes) 000870 1764791800.858518 mapwrite 0x00e313 thru 0x01663c (0x832a bytes) 000871 1764791800.944956 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000872 1764791801.104255 trunc from 0x029054 to 0x021826 (0x782d bytes) 000873 1764791801.133658 mapwrite 0x01e2b5 thru 0x02321e (0x4f6a bytes) 000874 1764791801.309780 mapread 0x016def thru 0x02321e (0xc430 bytes) 000876 1764791801.331846 read 0x009773 thru 0x00d507 (0x3d95 bytes) 000877 1764791801.349881 trunc from 0x02321f to 0x03d803 (0x1a5e5 bytes) 000878 1764791801.373102 trunc from 0x03d804 to 0x025532 (0x182d1 bytes) 000881 1764791801.401931 mapread 0x0088c6 thru 0x0115d5 (0x8d10 bytes) 000882 1764791801.415803 write 0x01bb31 thru 0x01c5c7 (0xa97 bytes) 000884 1764791801.417146 trunc from 0x025533 to 0x036b2d (0x115fb bytes) 000886 1764791801.451153 mapwrite 0x036252 thru 0x03cded (0x6b9c bytes) 000887 1764791801.553973 trunc from 0x03cdee to 0x0358a3 (0x754a bytes) 000888 1764791801.603580 mapwrite 0x033cde thru 0x03e148 (0xa46b bytes) 000889 1764791801.736892 trunc from 0x03e149 to 0x0387ad (0x599b bytes) 000897 1764791801.798547 mapread 0x000a10 thru 0x007998 (0x6f89 bytes) 000903 1764791801.809272 mapread 0x00c0f3 thru 0x015e67 (0x9d75 bytes) 000904 1764791801.818410 write 0x01ae41 thru 0x0268f7 (0xbab7 bytes) 000905 1764791801.823258 mapread 0x0068df thru 0x00a9cb (0x40ed bytes) 000907 1764791801.824423 mapread 0x0165a7 thru 0x02411c (0xdb76 bytes) 000909 1764791801.829856 MIRROR_EXTEND to 2 mirrors 000910 1764791802.801738 mapread 0x0371b5 thru 0x0387ad (0x15f9 bytes) 000911 1764791802.814733 trunc from 0x0387ae to 0x011f94 (0x26819 bytes) 000914 1764791802.911780 trunc from 0x011f95 to 0x01f911 (0xd97d bytes) 000917 1764791802.949939 mapwrite 0x02bca0 thru 0x02fcf9 (0x405a bytes) 000918 1764791803.019077 mapwrite 0x012f78 thru 0x01853f (0x55c8 bytes) 000919 1764791803.068155 trunc from 0x02fcfa to 0x02f84d (0x4ac bytes) 000921 1764791803.088601 MIRROR_EXTEND to 3 mirrors 000924 1764791803.826799 mapwrite 0x00f1a9 thru 0x01d045 (0xde9d bytes) 000926 1764791804.001865 mapread 0x00b989 thru 0x01b21e (0xf896 bytes) 000929 1764791804.003973 write 0x0128d4 thru 0x018ffe (0x672b bytes) 000931 1764791804.006496 mapread 0x00078f thru 0x0054c8 (0x4d3a bytes) 000934 1764791804.016301 mapread 0x00cf52 thru 0x010e0b (0x3eba bytes) 000936 1764791804.018179 mapwrite 0x004c45 thru 0x00c71f (0x7adb bytes) 000937 1764791804.060920 trunc from 0x02f84e to 0x00fcae (0x1fb9f bytes) 000938 1764791804.085989 read 0x00b905 thru 0x00fcae (0x43aa bytes) 000939 1764791804.090344 MIRROR_EXTEND to 4 mirrors 000940 1764791804.632851 mapread 0x004309 thru 0x004ce7 (0x9df bytes) 000941 1764791804.664622 MIRROR_EXTEND to 5 mirrors 000943 1764791805.303070 mapread 0x003cc1 thru 0x00f3a4 (0xb6e4 bytes) 000945 1764791805.336313 mapwrite 0x028e2f thru 0x02eb2b (0x5cfd bytes) 000946 1764791805.641688 read 0x00cd25 thru 0x01918c (0xc468 bytes) 000947 1764791805.657712 mapread 0x0044b8 thru 0x011efe (0xda47 bytes) 000950 1764791805.677333 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000952 1764791805.965489 mapwrite 0x001422 thru 0x00d3af (0xbf8e bytes) 000953 1764791806.041230 trunc from 0x02eb2c to 0x00bac4 (0x23067 bytes) 000954 1764791806.058653 read 0x0022ec thru 0x004983 (0x2698 bytes) 000956 1764791806.062537 write 0x0179b2 thru 0x018f22 (0x1571 bytes) 000958 1764791806.065955 mapwrite 0x00673e thru 0x00c0e4 (0x59a7 bytes) 000960 1764791806.174781 MIRROR_EXTEND to 5 mirrors 000961 1764791806.844692 write 0x032954 thru 0x03e0ed (0xb79a bytes) 000964 1764791806.970207 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000965 1764791807.213613 read 0x01db09 thru 0x0260e9 (0x85e1 bytes) 000966 1764791807.230806 MIRROR_RESYNC 4 mirrors 000968 1764791808.291001 write 0x02b88d thru 0x038c08 (0xd37c bytes) 000969 1764791808.353811 MIRROR_EXTEND to 5 mirrors 000970 1764791809.014709 read 0x0099ee thru 0x015749 (0xbd5c bytes) 000972 1764791809.049917 mapread 0x0157f3 thru 0x017ac0 (0x22ce bytes) 000973 1764791809.078980 MIRROR_RESYNC 5 mirrors 000975 1764791809.769578 write 0x00cecc thru 0x01ac6c (0xdda1 bytes) 000978 1764791809.848765 read 0x025f43 thru 0x02bda9 (0x5e67 bytes) 000979 1764791809.856392 mapread 0x01887a thru 0x01b3e9 (0x2b70 bytes) 000981 1764791809.865823 write 0x0221ea thru 0x02dc27 (0xba3e bytes) 000982 1764791809.876263 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x26f:0x0]: Invalid argument (22) 000983 1764791810.098525 trunc from 0x03e0ee to 0x03c552 (0x1b9b bytes) 000986 1764791810.119942 mapwrite 0x01048b thru 0x01c157 (0xbccd bytes) 000987 1764791810.259942 mapwrite 0x0110eb thru 0x01ae53 (0x9d69 bytes) 000988 1764791810.381689 trunc from 0x03c553 to 0x031e5d (0xa6f5 bytes) 000989 1764791810.440566 MIRROR_EXTEND to 5 mirrors 000990 1764791811.129902 read 0x02bc07 thru 0x031e5d (0x6257 bytes) 000991 1764791811.163628 mapread 0x006705 thru 0x00b72c (0x5028 bytes) 000992 1764791811.232862 trunc from 0x031e5e to 0x02ba83 (0x63da bytes) 000993 1764791811.388739 mapwrite 0x01583d thru 0x017e2b (0x25ef bytes) 000994 1764791811.447142 MIRROR_RESYNC 5 mirrors 000996 1764791812.878994 read 0x00a54a thru 0x00df92 (0x3a49 bytes) 000999 1764791812.889168 mapread 0x02987a thru 0x02ba83 (0x220a bytes) 001000 1764791812.912630 read 0x00075d thru 0x00af3f (0xa7e3 bytes) All operations completed A-OK! PASS 70a (84s) == sanity-flr test 71: check mirror extend parent fid ==== 14:56:58 (1764791818) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 200a: stress test ===================== 14:57:01 (1764791821) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4936896 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6596512 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6984992 .. Extending file size to 7802464 .. Extending file size to 8479392 .. done Extending file size to 8494240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8787392 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8636544 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8802560 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9153440 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8996416 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 55315 55316 55317 55319 55322 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) 192.168.204.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (74s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:58:15 (1764791895) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7195328 .. Extending file size to 1232480 .. done Extending file size to 7782720 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7685632 .. Extending file size to 8102240 .. Extending file size to 8776160 .. Extending file size to 9071616 .. Extending file size to 8348448 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8838848 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57181 57182 57184 57185 57186 done 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) 192.168.204.150@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:59:31 (1764791971) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4096 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4096 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143686 s, 14.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.405771 s, 5.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:59:48 (1764791988) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100292 s, 20.9 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 14:59:55 (1764791995) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2b9:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x398:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x398:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x399:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x393:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:00:02 (1764792002) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22369 s, 9.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x394:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x395:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:00:10 (1764792010) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23726 s, 8.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x396:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x396:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x397:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:00:18 (1764792018) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.271981 s, 7.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x398:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x398:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x399:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:00:53 (1764792053) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177699 s, 5.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x39f:0x0] } PASS 204d (61s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:01:54 (1764792114) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.388186 s, 5.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:02:57 (1764792177) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. Creating new pool test_204f oleg450-server: Pool lustre.test_204f created Adding targets to pool oleg450-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg450-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28106 s, 7.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x39e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg450-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg450-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg450-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:04:18 (1764792258) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:04:26 (1764792266) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:04:33 (1764792273) oleg450-server: Pool lustre.test_206 created Adding targets to pool oleg450-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg450-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg450-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg450-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg450-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:04:58 (1764792298) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560777 s, 18.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338813 s, 30.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfd98beeb Mirror 3: 0xfd98beeb PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:05:07 (1764792307) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:05:11 (1764792311) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:05:14 (1764792314) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.139773 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x11c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2100000 Read this (4 bytes): BBBB 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:05:48 (1764792348) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.024995 s, 328 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:05:55 (1764792355) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658551 s, 15.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2de lmm_fid: [0x200000404:0x2de:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2de lmm_fid: [0x200000404:0x2de:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x3ac:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834654 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-12-03 15:05:57.000000000 -0500 Modify: 2025-12-03 15:05:57.000000000 -0500 Change: 2025-12-03 15:05:59.000000000 -0500 Birth: 2025-12-03 15:05:56.000000000 -0500 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:06:03 (1764792363) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:06:06 (1764792366) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.758114 s, 54.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0381403 s, 107 kB/s size after second write -rw-r--r-- 1 root root 4096 Dec 3 15:06 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2e0:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Dec 3 15:06 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test complete, duration 1325 sec =========== 15:06:16 (1764792376) === sanity-flr: start cleanup 15:06:18 (1764792378) === === sanity-flr: finish cleanup 15:06:24 (1764792384) ===