-----============= acceptance-small: sanity-flr ============----- Wed Aug 13 10:39:52 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg205-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 10:40:00 (1755096000) === oleg205-client.virtnet: executing check_config_client /mnt/lustre oleg205-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg205-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b9ec9055800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b9ec9055800.idle_timeout=debug disable quota as required oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 10:40:15 (1755096015) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7562] == sanity-flr test 0a: lfs mirror create with -N option == 10:40:16 (1755096016) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:40:22 (1755096022) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:40:24 (1755096024) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:40:25 (1755096025) PASS 0d (11s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:40:36 (1755096036) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:40:37 (1755096037) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 10:40:39 (1755096039) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8b9ec9055800.stats=clear osc.lustre-OST0001-osc-ffff8b9ec9055800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:40:42 (1755096042) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:40:47 (1755096047) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:40:50 (1755096050) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:40:52 (1755096052) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 10:40:56 (1755096056) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:40:59 (1755096059) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:41:00 (1755096060) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:41:04 (1755096064) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:41:09 (1755096069) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:41:13 (1755096073) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.476783 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882793 s, 11.9 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:41:18 (1755096078) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.149747 s, 98.0 MB/s PASS 22 (4s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:41:22 (1755096082) Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b9ec9055800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b9ec9055800.ost_server_uuid in DISCONN state after 1 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b9ec9055800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b9ec9055800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (33s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:41:55 (1755096115) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.248446 s, 46.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b9ec9055800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b9ec9055800.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (18s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:42:13 (1755096133) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b9ec9055800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b9ec9055800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b9ec9055800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b9ec9055800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (32s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:42:45 (1755096165) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.16415 s, 45.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 13 10:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.446s user 0m0.000s sys 0m0.185s Stopping /mnt/lustre-ost1 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b9ec9055800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b9ec9055800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.609s user 0m0.000s sys 0m0.189s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg205-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg205-server oleg205-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b9ec9055800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b9ec9055800.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost1)... real 0m0.517s user 0m0.000s sys 0m0.192s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg205-server: oleg205-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg205-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg205-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b9ec9055800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b9ec9055800.ost_server_uuid in FULL state after 0 sec PASS 33b (39s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:43:24 (1755096204) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:43:26 (1755096206) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:43:28 (1755096208) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 10:43:29 (1755096209) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 10:43:33 (1755096213) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.280927 s, 44.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:43:39 (1755096219) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156775 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170471 s, 61.5 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:43:42 (1755096222) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:43:46 (1755096226) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000719348 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00069656 s, 1.5 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000886598 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00134565 s, 761 kB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 10:43:51 (1755096231) create test files with size 15744427 .. 9a5e78e3ca1ee5a4c10c4950923487de - 8ca3b476fe1108d73cd6850eee2ecc5a - 0f5084dab42c4b700ded53c7524346b6 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre PASS 37 (11s) == sanity-flr test 38: resync ============================ 10:44:02 (1755096242) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:44:09 (1755096249) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:44:12 (1755096252) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052812 s, 19.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143034 s, 73.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 10:44:16 (1755096256) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0452026 s, 46.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.127258 s, 33.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0708349 s, 29.6 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 10:44:20 (1755096260) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 10:44:22 (1755096262) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:44:23 (1755096263) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129958 s, 80.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 10:44:26 (1755096266) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (2s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:44:28 (1755096268) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:44:32 (1755096272) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0990005 s, 106 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (18s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:44:50 (1755096290) llite.lustre-ffff8b9ec8e8d000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0971209 s, 108 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg205-client.virtnet /mnt/lustre (opts:) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8b9ed1515000.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:44:55 (1755096295) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (16s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:45:11 (1755096311) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:45:26 (1755096326) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 10:45:27 (1755096327) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0727539 s, 43.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0618499 s, 50.9 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 7887a70be26ec9cc47cff62640a8565d - mirror 1 checksum: 3259f18134e5ba33da109836f8b9db96 - mirror 2 checksum: 7887a70be26ec9cc47cff62640a8565d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Aug 13 10:45:29 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Wed Aug 13 10:45:29 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7887a70be26ec9cc47cff62640a8565d - mirror 1 checksum: 3259f18134e5ba33da109836f8b9db96 - mirror 2 checksum: 3259f18134e5ba33da109836f8b9db96 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Aug 13 10:45:29 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Wed Aug 13 10:45:29 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:45:31 (1755096331) Failing mds1 on oleg205-server Stopping /mnt/lustre-mds1 (opts:) on oleg205-server 10:45:34 (1755096334) shut down facet: mds1 facet_host: oleg205-server facet_failover_host: oleg205-server Failover mds1 to oleg205-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg205-client: oleg205-server: ssh exited with exit code 1 Started lustre-MDT0000 10:45:49 (1755096349) targets are mounted 10:45:49 (1755096349) facet_failover done oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127593 s, 82.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (25s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:45:56 (1755096356) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:45:57 (1755096357) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:45:59 (1755096359) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00378388 s, 4.3 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:46:01 (1755096361) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:46:03 (1755096363) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:46:04 (1755096364) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (7s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:46:11 (1755096371) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:46:31 (1755096391) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:46:49 (1755096409) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:47:07 (1755096427) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 10:47:10 (1755096430) mirror create pid 52250 mirror split pid 52253 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 52250 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 52253 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (64s) == sanity-flr test 70a: flr mode fsx test ================ 10:48:14 (1755096494) Seed set to 6495 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1755096495.177742 mapwrite 0x020d34 thru 0x02cd5a (0xc027 bytes) 000002 1755096495.290889 write 0x016289 thru 0x01bb33 (0x58ab bytes) 000004 1755096495.300532 mapread 0x00ffe0 thru 0x01c86e (0xc88f bytes) 000006 1755096495.302391 write 0x016734 thru 0x0170e7 (0x9b4 bytes) 000008 1755096495.303790 write 0x01322e thru 0x01f43e (0xc211 bytes) 000009 1755096495.305010 mapwrite 0x0351fb thru 0x03ffff (0xae05 bytes) 000012 1755096495.367048 MIRROR_EXTEND to 2 mirrors 000013 1755096495.732651 MIRROR_EXTEND to 3 mirrors truncating to largest ever: 0xd3c7 000014 1755096496.041424 trunc from 0x040000 to 0x00d3c6 (0x32c39 bytes) 000015 1755096496.114776 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000016 1755096496.229927 mapwrite 0x001f01 thru 0x00c145 (0xa245 bytes) 000019 1755096496.275515 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) truncating to largest ever: 0x3206b 000020 1755096496.359934 trunc from 0x00d3c7 to 0x03206a (0x24ca4 bytes) 000021 1755096496.376055 trunc from 0x03206b to 0x024f49 (0xd121 bytes) 000022 1755096496.387409 mapread 0x00a3b7 thru 0x01a04e (0xfc98 bytes) 000026 1755096496.408317 read 0x007727 thru 0x0133dd (0xbcb7 bytes) 000032 1755096496.411595 mapwrite 0x011974 thru 0x01b322 (0x99af bytes) 000034 1755096496.452155 mapread 0x005a90 thru 0x009452 (0x39c3 bytes) 000036 1755096496.453591 write 0x0217a0 thru 0x02f0f6 (0xd957 bytes) 000038 1755096496.466167 MIRROR_EXTEND to 2 mirrors 000046 1755096496.734119 mapread 0x02256f thru 0x02c730 (0xa1c2 bytes) 000047 1755096496.745393 read 0x0192ea thru 0x01c29d (0x2fb4 bytes) 000048 1755096496.750060 mapread 0x02dd44 thru 0x02f0f6 (0x13b3 bytes) 000050 1755096496.760737 mapread 0x02c6d6 thru 0x02ee9c (0x27c7 bytes) 000053 1755096496.762243 read 0x01b949 thru 0x0292c0 (0xd978 bytes) 000055 1755096496.762932 mapwrite 0x017092 thru 0x017412 (0x381 bytes) 000058 1755096496.877463 read 0x00a13e thru 0x00ba55 (0x1918 bytes) 000059 1755096496.886811 mapread 0x0298d1 thru 0x02f0f6 (0x5826 bytes) 000060 1755096496.887732 read 0x00548e thru 0x00acb2 (0x5825 bytes) 000062 1755096496.888665 read 0x003553 thru 0x00ef8b (0xba39 bytes) 000064 1755096496.889622 write 0x02bc1e thru 0x039c2e (0xe011 bytes) 000066 1755096496.891339 read 0x009706 thru 0x00e10e (0x4a09 bytes) 000067 1755096496.895951 MIRROR_RESYNC 2 mirrors 000069 1755096497.186511 mapread 0x02273a thru 0x026e10 (0x46d7 bytes) 000070 1755096497.202061 mapread 0x024b96 thru 0x029643 (0x4aae bytes) 000072 1755096497.202960 write 0x005870 thru 0x014825 (0xefb6 bytes) 000073 1755096497.257687 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000074 1755096497.341016 trunc from 0x039c2f to 0x0225bf (0x1766f bytes) 000075 1755096497.352476 mapread 0x00019f thru 0x005513 (0x5375 bytes) 000077 1755096497.366988 mapread 0x014c58 thru 0x019e3f (0x51e8 bytes) 000082 1755096497.375198 mapread 0x012c7a thru 0x01c2cc (0x9653 bytes) 000083 1755096497.376179 write 0x030289 thru 0x03bab2 (0xb82a bytes) 000084 1755096497.377439 read 0x012f5d thru 0x013663 (0x707 bytes) 000086 1755096497.378671 read 0x01d6d2 thru 0x01de39 (0x768 bytes) 000088 1755096497.384730 mapwrite 0x000f1f thru 0x003c48 (0x2d2a bytes) 000089 1755096497.414985 trunc from 0x03bab3 to 0x01235f (0x29753 bytes) 000090 1755096497.424144 write 0x03258c thru 0x033d86 (0x17fb bytes) 000093 1755096497.425849 read 0x019166 thru 0x027970 (0xe80b bytes) 000094 1755096497.432915 mapread 0x028cd3 thru 0x033d86 (0xb0b4 bytes) 000097 1755096497.434740 read 0x016bd7 thru 0x020328 (0x9752 bytes) 000098 1755096497.435333 mapread 0x0061c5 thru 0x00811b (0x1f57 bytes) 000099 1755096497.436250 trunc from 0x033d87 to 0x006e29 (0x2cf5d bytes) 000101 1755096497.449521 write 0x00a4bb thru 0x00f398 (0x4ede bytes) truncating to largest ever: 0x3feed 000103 1755096497.451161 trunc from 0x00f399 to 0x03feec (0x30b54 bytes) 000104 1755096497.469804 mapread 0x010d2d thru 0x01bca2 (0xaf76 bytes) 000106 1755096497.481732 read 0x03d1f5 thru 0x03e062 (0xe6e bytes) 000107 1755096497.482333 mapwrite 0x032241 thru 0x0339e3 (0x17a3 bytes) 000109 1755096497.528760 read 0x03582d thru 0x03ec4a (0x941e bytes) 000110 1755096497.529459 trunc from 0x03feed to 0x03e87d (0x166f bytes) 000112 1755096497.543583 trunc from 0x03e87e to 0x02f0b9 (0xf7c4 bytes) 000114 1755096497.557000 mapwrite 0x026cc8 thru 0x02f618 (0x8951 bytes) 000115 1755096497.605651 trunc from 0x02f619 to 0x01238a (0x1d28e bytes) 000118 1755096497.620108 mapwrite 0x023aeb thru 0x02beb6 (0x83cc bytes) 000120 1755096497.679333 write 0x0170c9 thru 0x01d0d4 (0x600c bytes) 000121 1755096497.680142 trunc from 0x02beb7 to 0x03d47e (0x115c8 bytes) 000122 1755096497.687940 mapread 0x013501 thru 0x01a52b (0x702b bytes) 000123 1755096497.688856 read 0x025705 thru 0x02cc62 (0x755e bytes) 000124 1755096497.700465 trunc from 0x03d47f to 0x01eafa (0x1e984 bytes) 000125 1755096497.714331 trunc from 0x01eafb to 0x03911b (0x1a621 bytes) 000126 1755096497.723205 write 0x00988d thru 0x00cad1 (0x3245 bytes) 000128 1755096497.724455 trunc from 0x03911c to 0x00c71c (0x2c9ff bytes) 000129 1755096497.735335 write 0x02bce4 thru 0x038d55 (0xd072 bytes) 000130 1755096497.736977 mapread 0x00e63a thru 0x00efe6 (0x9ad bytes) 000131 1755096497.742655 mapwrite 0x012733 thru 0x016342 (0x3c10 bytes) 000133 1755096497.803914 write 0x01da0f thru 0x025940 (0x7f32 bytes) 000136 1755096497.816980 mapwrite 0x013dcc thru 0x017cd2 (0x3f07 bytes) 000137 1755096497.854496 MIRROR_EXTEND to 2 mirrors 000139 1755096498.190217 mapread 0x01a255 thru 0x0207c4 (0x6570 bytes) 000143 1755096498.209164 MIRROR_RESYNC 2 mirrors 000144 1755096498.283102 trunc from 0x038d56 to 0x0265e8 (0x1276d bytes) 000145 1755096498.319210 trunc from 0x0265e9 to 0x010bb1 (0x15a37 bytes) 000146 1755096498.329982 write 0x004599 thru 0x007978 (0x33e0 bytes) 000148 1755096498.336313 read 0x009883 thru 0x010bb1 (0x732f bytes) 000150 1755096498.347203 write 0x00973c thru 0x014de2 (0xb6a7 bytes) 000152 1755096498.349156 mapwrite 0x023c16 thru 0x02a465 (0x6850 bytes) 000153 1755096498.432074 write 0x025501 thru 0x02fdc6 (0xa8c6 bytes) 000154 1755096498.433373 trunc from 0x02fdc7 to 0x031bba (0x1df4 bytes) 000157 1755096498.441938 mapwrite 0x00751a thru 0x0155cd (0xe0b4 bytes) 000158 1755096498.493470 read 0x001c26 thru 0x01161d (0xf9f8 bytes) 000159 1755096498.494075 read 0x02c530 thru 0x02ecfa (0x27cb bytes) 000160 1755096498.494746 mapwrite 0x028711 thru 0x033524 (0xae14 bytes) 000161 1755096498.545968 write 0x00107a thru 0x00634f (0x52d6 bytes) 000163 1755096498.550650 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000164 1755096498.651779 write 0x005e8d thru 0x006fa2 (0x1116 bytes) 000165 1755096498.660401 write 0x020c56 thru 0x02ba67 (0xae12 bytes) 000167 1755096498.672738 write 0x020001 thru 0x020062 (0x62 bytes) 000168 1755096498.673351 trunc from 0x033525 to 0x01b5c8 (0x17f5c bytes) 000169 1755096498.683782 mapwrite 0x03b668 thru 0x03ffff (0x4998 bytes) 000170 1755096498.729151 read 0x0051c1 thru 0x00832a (0x316a bytes) 000171 1755096498.732884 read 0x00ce03 thru 0x00f2df (0x24dd bytes) 000173 1755096498.741795 read 0x0003a2 thru 0x001954 (0x15b3 bytes) 000174 1755096498.742258 read 0x00471a thru 0x01370c (0xeff3 bytes) 000175 1755096498.742845 trunc from 0x040000 to 0x02e22f (0x11dd0 bytes) 000178 1755096498.760934 mapwrite 0x031df0 thru 0x038c40 (0x6e51 bytes) 000183 1755096498.810679 trunc from 0x038c41 to 0x00a81a (0x2e426 bytes) 000187 1755096498.822120 write 0x0000dc thru 0x002019 (0x1f3e bytes) 000190 1755096498.823976 write 0x01f0ff thru 0x021459 (0x235b bytes) 000192 1755096498.826201 mapwrite 0x018826 thru 0x01bff3 (0x37ce bytes) 000193 1755096498.867509 mapwrite 0x0005a9 thru 0x00bc9d (0xb6f5 bytes) 000194 1755096498.893455 write 0x02eef7 thru 0x031fc4 (0x30ce bytes) 000195 1755096498.894967 read 0x0272fb thru 0x031fc4 (0xacca bytes) 000197 1755096498.901497 trunc from 0x031fc5 to 0x0314aa (0xb1a bytes) 000198 1755096498.914365 trunc from 0x0314ab to 0x0237c5 (0xdce5 bytes) 000199 1755096498.923630 mapwrite 0x013091 thru 0x019fb2 (0x6f22 bytes) 000200 1755096498.951440 write 0x034385 thru 0x03a141 (0x5dbd bytes) 000201 1755096498.952645 read 0x005709 thru 0x00b4ea (0x5de2 bytes) 000202 1755096498.953206 write 0x0047eb thru 0x0147be (0xffd4 bytes) 000204 1755096498.954896 write 0x00bfef thru 0x0171bb (0xb1cd bytes) 000205 1755096498.956106 write 0x00967c thru 0x00a34e (0xcd3 bytes) 000208 1755096498.957699 read 0x00a67c thru 0x01031d (0x5ca2 bytes) 000209 1755096498.958182 trunc from 0x03a142 to 0x020c79 (0x194c8 bytes) 000211 1755096498.968038 read 0x0032b7 thru 0x011862 (0xe5ac bytes) 000213 1755096498.969104 mapwrite 0x036f79 thru 0x03db45 (0x6bcd bytes) 000214 1755096499.012798 trunc from 0x03db46 to 0x02f69f (0xe4a6 bytes) 000215 1755096499.022366 mapwrite 0x01979e thru 0x024387 (0xabea bytes) 000222 1755096499.048411 write 0x031000 thru 0x03f0b0 (0xe0b1 bytes) 000223 1755096499.049840 mapread 0x014c21 thru 0x022e74 (0xe254 bytes) 000224 1755096499.050679 mapwrite 0x023f91 thru 0x033aea (0xfb5a bytes) 000226 1755096499.085219 mapwrite 0x02c19b thru 0x033a7d (0x78e3 bytes) 000227 1755096499.109586 write 0x03f3e7 thru 0x03ffff (0xc19 bytes) 000230 1755096499.114061 MIRROR_EXTEND to 2 mirrors 000232 1755096499.320804 trunc from 0x040000 to 0x03f741 (0x8be bytes) 000235 1755096499.365707 trunc from 0x03f742 to 0x022cfd (0x1ca44 bytes) 000236 1755096499.380543 write 0x02a4bd thru 0x035cc8 (0xb80c bytes) 000237 1755096499.382054 write 0x001bcb thru 0x008474 (0x68aa bytes) 000240 1755096499.396577 read 0x0125a2 thru 0x018205 (0x5c64 bytes) 000243 1755096499.406663 write 0x02cb01 thru 0x03a353 (0xd853 bytes) 000245 1755096499.412240 mapread 0x0081bc thru 0x00fcf3 (0x7b38 bytes) 000246 1755096499.422808 mapwrite 0x03607e thru 0x03ffff (0x9f82 bytes) 000249 1755096499.490538 mapwrite 0x00426e thru 0x010bf8 (0xc98b bytes) 000252 1755096499.525137 trunc from 0x040000 to 0x03bb80 (0x447f bytes) 000254 1755096499.539620 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000255 1755096499.620755 mapwrite 0x016b58 thru 0x01956a (0x2a13 bytes) 000256 1755096499.659236 mapwrite 0x03c784 thru 0x03fb1f (0x339c bytes) 000257 1755096499.702056 trunc from 0x03fb20 to 0x032fda (0xcb45 bytes) 000259 1755096499.716056 mapread 0x01ad58 thru 0x01dacf (0x2d78 bytes) 000261 1755096499.717410 write 0x022c26 thru 0x02f18a (0xc565 bytes) 000262 1755096499.725380 mapread 0x00bd0a thru 0x011ceb (0x5fe2 bytes) 000263 1755096499.731133 mapwrite 0x016a8f thru 0x017287 (0x7f9 bytes) 000264 1755096499.766187 mapwrite 0x02e418 thru 0x0374ad (0x9096 bytes) 000268 1755096499.814773 trunc from 0x0374ae to 0x001e52 (0x3565b bytes) 000269 1755096499.824102 mapread 0x001956 thru 0x001e52 (0x4fd bytes) 000270 1755096499.830372 MIRROR_EXTEND to 2 mirrors 000271 1755096499.997636 write 0x00b2ca thru 0x012ec3 (0x7bfa bytes) 000275 1755096500.029489 write 0x0089db thru 0x015710 (0xcd36 bytes) 000276 1755096500.033764 trunc from 0x015711 to 0x01de84 (0x8774 bytes) 000277 1755096500.056768 trunc from 0x01de85 to 0x02ae98 (0xd014 bytes) 000279 1755096500.065450 trunc from 0x02ae99 to 0x014ca8 (0x161f0 bytes) 000281 1755096500.079083 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000283 1755096500.186808 read 0x0106da thru 0x010f13 (0x83a bytes) 000285 1755096500.191907 mapread 0x003d20 thru 0x00700b (0x32ec bytes) 000286 1755096500.207890 trunc from 0x014ca9 to 0x030913 (0x1bc6b bytes) 000287 1755096500.223871 trunc from 0x030914 to 0x011e55 (0x1eabe bytes) 000288 1755096500.237155 write 0x03d56c thru 0x03ffff (0x2a94 bytes) 000289 1755096500.238158 mapwrite 0x00f782 thru 0x01a017 (0xa896 bytes) 000291 1755096500.282646 write 0x009704 thru 0x010979 (0x7276 bytes) 000293 1755096500.284037 mapread 0x016036 thru 0x01adef (0x4dba bytes) 000295 1755096500.285238 read 0x0117d6 thru 0x01e186 (0xc9b1 bytes) 000297 1755096500.286639 read 0x0025b4 thru 0x0107ae (0xe1fb bytes) 000298 1755096500.287182 trunc from 0x040000 to 0x023412 (0x1cbed bytes) 000299 1755096500.296996 mapread 0x011106 thru 0x017ca5 (0x6ba0 bytes) 000300 1755096500.297806 read 0x019e80 thru 0x023412 (0x9593 bytes) 000302 1755096500.303914 mapwrite 0x02c99a thru 0x02d0bb (0x722 bytes) 000303 1755096500.349973 mapread 0x014d92 thru 0x022d9a (0xe009 bytes) 000304 1755096500.350923 write 0x02a4dc thru 0x037d16 (0xd83b bytes) 000305 1755096500.352173 mapread 0x02b3b8 thru 0x03405a (0x8ca3 bytes) 000306 1755096500.353035 mapwrite 0x0189cc thru 0x01ae82 (0x24b7 bytes) 000307 1755096500.375913 write 0x0028f5 thru 0x01194f (0xf05b bytes) 000308 1755096500.377244 mapread 0x01eb33 thru 0x0242db (0x57a9 bytes) 000309 1755096500.378079 mapread 0x001bb1 thru 0x00bf6d (0xa3bd bytes) 000311 1755096500.384591 MIRROR_EXTEND to 2 mirrors 000312 1755096500.554005 read 0x01c00c thru 0x029789 (0xd77e bytes) 000314 1755096500.559117 read 0x003313 thru 0x00f6ba (0xc3a8 bytes) 000315 1755096500.564760 trunc from 0x037d17 to 0x012047 (0x25ccf bytes) 000316 1755096500.605529 read 0x001a7d thru 0x0064a0 (0x4a24 bytes) 000317 1755096500.611432 trunc from 0x012048 to 0x01c18f (0xa148 bytes) 000318 1755096500.619991 trunc from 0x01c190 to 0x0115f3 (0xab9c bytes) 000320 1755096500.626781 write 0x030fe2 thru 0x03f72f (0xe74e bytes) 000322 1755096500.628753 mapwrite 0x029c7f thru 0x02a3fc (0x77e bytes) 000326 1755096500.656572 mapwrite 0x035c8c thru 0x03e4bc (0x8831 bytes) 000328 1755096500.678386 write 0x029f62 thru 0x02ae7a (0xf19 bytes) 000330 1755096500.679562 read 0x0326b2 thru 0x032999 (0x2e8 bytes) 000332 1755096500.680419 mapread 0x013b0f thru 0x01a863 (0x6d55 bytes) 000334 1755096500.681501 trunc from 0x03f730 to 0x0130a7 (0x2c688 bytes) 000335 1755096500.691126 MIRROR_RESYNC 2 mirrors 000336 1755096500.854451 read 0x00a3ba thru 0x0130a7 (0x8cee bytes) 000339 1755096500.860228 mapwrite 0x038bde thru 0x03ffff (0x7422 bytes) 000340 1755096500.943605 MIRROR_EXTEND to 3 mirrors 000342 1755096501.214066 trunc from 0x040000 to 0x0166ec (0x29913 bytes) 000343 1755096501.250182 mapwrite 0x007cc5 thru 0x0164a0 (0xe7dc bytes) 000345 1755096501.310387 MIRROR_RESYNC 3 mirrors 000350 1755096501.531328 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000353 1755096501.632662 read 0x012ca2 thru 0x0166ec (0x3a4b bytes) 000354 1755096501.638806 read 0x00f81a thru 0x0166ec (0x6ed3 bytes) 000355 1755096501.650473 MIRROR_EXTEND to 3 mirrors 000356 1755096501.863899 trunc from 0x0166ed to 0x01255b (0x4191 bytes) 000358 1755096501.910543 write 0x007c61 thru 0x0155bd (0xd95d bytes) 000359 1755096501.914513 mapread 0x009118 thru 0x0155bd (0xc4a6 bytes) 000361 1755096501.915858 write 0x025690 thru 0x033536 (0xdea7 bytes) 000362 1755096501.917440 trunc from 0x033537 to 0x026f58 (0xc5de bytes) 000363 1755096501.927254 trunc from 0x026f59 to 0x03b29f (0x14347 bytes) 000364 1755096501.935955 write 0x03dd4a thru 0x03ffff (0x22b6 bytes) 000365 1755096501.937017 mapwrite 0x01eff5 thru 0x022505 (0x3511 bytes) 000367 1755096501.970437 mapwrite 0x01c13b thru 0x01e22c (0x20f2 bytes) 000368 1755096501.990579 write 0x02d429 thru 0x02f727 (0x22ff bytes) 000370 1755096501.996150 write 0x02554e thru 0x02833d (0x2df0 bytes) 000371 1755096501.999594 MIRROR_EXTEND to 4 mirrors 000372 1755096502.243959 mapread 0x03f9ee thru 0x03ffff (0x612 bytes) 000373 1755096502.252949 mapwrite 0x0039cd thru 0x00d847 (0x9e7b bytes) 000374 1755096502.343979 MIRROR_RESYNC 4 mirrors 000375 1755096502.676867 trunc from 0x040000 to 0x0385dd (0x7a22 bytes) 000378 1755096502.707411 trunc from 0x0385de to 0x038707 (0x12a bytes) 000380 1755096502.717810 mapwrite 0x037b67 thru 0x038c2c (0x10c6 bytes) 000381 1755096502.767520 mapwrite 0x0347ea thru 0x03ffff (0xb816 bytes) 000382 1755096502.822861 mapwrite 0x006de9 thru 0x01098c (0x9ba4 bytes) 000384 1755096502.854430 read 0x003679 thru 0x00cc3b (0x95c3 bytes) 000385 1755096502.855434 read 0x01fc18 thru 0x0295e5 (0x99ce bytes) 000388 1755096502.861567 trunc from 0x040000 to 0x025a3e (0x1a5c1 bytes) 000389 1755096502.869480 read 0x01a589 thru 0x01c8d4 (0x234c bytes) 000390 1755096502.870109 write 0x0072ac thru 0x00eef7 (0x7c4c bytes) 000391 1755096502.871045 mapwrite 0x01ad49 thru 0x023b0f (0x8dc7 bytes) 000392 1755096502.903067 mapwrite 0x0135e3 thru 0x0186df (0x50fd bytes) 000396 1755096502.935560 trunc from 0x025a3f to 0x029266 (0x3828 bytes) 000397 1755096502.944134 write 0x00a47f thru 0x00d205 (0x2d87 bytes) 000398 1755096502.945066 mapread 0x016f48 thru 0x0212af (0xa368 bytes) 000399 1755096502.945941 write 0x03ac6b thru 0x03ffff (0x5395 bytes) 000401 1755096502.947511 mapread 0x01c3d7 thru 0x02b137 (0xed61 bytes) 000402 1755096502.956819 mapwrite 0x001aaa thru 0x009e24 (0x837b bytes) 000403 1755096502.992085 mapwrite 0x03e17b thru 0x03ffff (0x1e85 bytes) 000404 1755096503.025065 mapwrite 0x01f339 thru 0x02c9da (0xd6a2 bytes) 000405 1755096503.058170 read 0x018d0e thru 0x01d234 (0x4527 bytes) 000406 1755096503.058773 write 0x0082dc thru 0x00b4e5 (0x320a bytes) 000407 1755096503.059697 read 0x00a4b5 thru 0x00ab83 (0x6cf bytes) 000408 1755096503.060196 trunc from 0x040000 to 0x0313b3 (0xec4c bytes) 000410 1755096503.069920 mapwrite 0x0132a8 thru 0x01bcca (0x8a23 bytes) 000411 1755096503.103816 trunc from 0x0313b4 to 0x01b130 (0x16283 bytes) 000412 1755096503.116031 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000413 1755096503.210891 trunc from 0x01b131 to 0x034698 (0x19568 bytes) 000416 1755096503.223119 MIRROR_RESYNC 3 mirrors 000418 1755096503.487079 mapwrite 0x02aa11 thru 0x02de4e (0x343e bytes) 000419 1755096503.564591 mapwrite 0x0390f1 thru 0x03ffff (0x6f0f bytes) 000420 1755096503.606549 read 0x02fa29 thru 0x039094 (0x966c bytes) 000421 1755096503.607153 trunc from 0x040000 to 0x0056bd (0x3a942 bytes) 000424 1755096503.616626 trunc from 0x0056be to 0x013001 (0xd944 bytes) 000427 1755096503.626276 trunc from 0x013002 to 0x03b539 (0x28538 bytes) 000428 1755096503.634808 write 0x001c02 thru 0x00f68b (0xda8a bytes) 000429 1755096503.642224 trunc from 0x03b53a to 0x016729 (0x24e10 bytes) 000430 1755096503.653844 MIRROR_EXTEND to 4 mirrors 000431 1755096503.869513 trunc from 0x01672a to 0x0125da (0x414f bytes) 000434 1755096503.905722 write 0x032b28 thru 0x0381ea (0x56c3 bytes) 000440 1755096503.910202 write 0x0128c9 thru 0x0217d9 (0xef11 bytes) 000441 1755096503.917542 write 0x03bc73 thru 0x03d0dd (0x146b bytes) 000444 1755096503.924824 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000445 1755096504.040273 write 0x03577f thru 0x03de46 (0x86c8 bytes) 000446 1755096504.052737 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000448 1755096504.181424 trunc from 0x03de47 to 0x037e28 (0x601e bytes) 000449 1755096504.189711 mapwrite 0x001066 thru 0x002a61 (0x19fc bytes) 000452 1755096504.221167 mapread 0x006338 thru 0x009d62 (0x3a2b bytes) 000453 1755096504.227064 read 0x00e53e thru 0x019cc4 (0xb787 bytes) 000454 1755096504.232803 mapwrite 0x03cc67 thru 0x03f8d9 (0x2c73 bytes) 000456 1755096504.275164 write 0x03e7d0 thru 0x03ffff (0x1830 bytes) 000457 1755096504.276023 mapread 0x0199b1 thru 0x01e61e (0x4c6e bytes) 000458 1755096504.282526 write 0x02df26 thru 0x02fc67 (0x1d42 bytes) 000459 1755096504.289493 trunc from 0x040000 to 0x037299 (0x8d66 bytes) 000460 1755096504.301471 trunc from 0x03729a to 0x03dc38 (0x699f bytes) 000461 1755096504.314445 MIRROR_EXTEND to 3 mirrors 000462 1755096504.515441 trunc from 0x03dc39 to 0x002987 (0x3b2b1 bytes) 000464 1755096504.563897 write 0x01241a thru 0x014588 (0x216f bytes) 000465 1755096504.564959 read 0x0034f2 thru 0x01148c (0xdf9b bytes) 000466 1755096504.571539 mapread 0x0026ba thru 0x00a699 (0x7fe0 bytes) 000469 1755096504.576337 read 0x00e689 thru 0x0107cd (0x2145 bytes) 000470 1755096504.576926 mapread 0x0120e9 thru 0x014588 (0x24a0 bytes) 000471 1755096504.577696 mapread 0x00e225 thru 0x014588 (0x6364 bytes) 000473 1755096504.579000 write 0x019ac9 thru 0x020699 (0x6bd1 bytes) 000475 1755096504.580716 write 0x004c86 thru 0x007cc1 (0x303c bytes) 000480 1755096504.586040 MIRROR_EXTEND to 4 mirrors 000482 1755096504.781536 mapread 0x003808 thru 0x012182 (0xe97b bytes) 000484 1755096504.799827 write 0x008350 thru 0x01282d (0xa4de bytes) 000485 1755096504.830342 mapread 0x014a72 thru 0x01c386 (0x7915 bytes) 000486 1755096504.834819 mapread 0x00f470 thru 0x01c2ea (0xce7b bytes) 000487 1755096504.835643 mapwrite 0x00b43c thru 0x0166ff (0xb2c4 bytes) 000489 1755096504.864759 mapwrite 0x012dcb thru 0x022972 (0xfba8 bytes) 000491 1755096504.929221 read 0x003ce8 thru 0x00ecc6 (0xafdf bytes) 000493 1755096504.930161 mapread 0x008339 thru 0x00c1d9 (0x3ea1 bytes) 000495 1755096504.931337 read 0x0031a3 thru 0x008238 (0x5096 bytes) 000496 1755096504.931892 trunc from 0x022973 to 0x031ccf (0xf35d bytes) 000498 1755096504.943951 MIRROR_RESYNC 4 mirrors 000499 1755096505.279164 write 0x03238d thru 0x03b5a5 (0x9219 bytes) 000501 1755096505.311756 mapwrite 0x0222bf thru 0x0230e4 (0xe26 bytes) 000502 1755096505.342380 write 0x008889 thru 0x013dd3 (0xb54b bytes) 000504 1755096505.354505 trunc from 0x03b5a6 to 0x00bf1a (0x2f68b bytes) 000506 1755096505.368019 read 0x006c7f thru 0x00bf1a (0x529c bytes) 000508 1755096505.368802 mapread 0x002d5a thru 0x00bf1a (0x91c1 bytes) 000509 1755096505.369634 mapwrite 0x01af98 thru 0x021a31 (0x6a9a bytes) 000512 1755096505.406996 mapwrite 0x008316 thru 0x013808 (0xb4f3 bytes) 000515 1755096505.440598 MIRROR_RESYNC 4 mirrors 000516 1755096505.749238 MIRROR_RESYNC 4 mirrors 000517 1755096505.824929 mapwrite 0x02a3c3 thru 0x0357bc (0xb3fa bytes) 000518 1755096505.915971 trunc from 0x0357bd to 0x014e77 (0x20945 bytes) 000520 1755096505.926871 read 0x007d90 thru 0x00ce70 (0x50e1 bytes) 000521 1755096505.931135 mapread 0x00c67e thru 0x014e77 (0x87fa bytes) 000522 1755096505.937379 mapread 0x00ee91 thru 0x014e77 (0x5fe7 bytes) 000524 1755096505.938686 write 0x03fafd thru 0x03ffff (0x503 bytes) 000527 1755096505.940386 mapwrite 0x019436 thru 0x027d2e (0xe8f9 bytes) 000529 1755096505.976072 mapread 0x02cc68 thru 0x02ea2a (0x1dc3 bytes) 000531 1755096505.981574 trunc from 0x040000 to 0x02ab3f (0x154c0 bytes) 000532 1755096505.989428 read 0x005506 thru 0x008c4c (0x3747 bytes) 000533 1755096505.994106 trunc from 0x02ab40 to 0x005d33 (0x24e0c bytes) 000535 1755096506.003251 write 0x02f934 thru 0x03b0be (0xb78b bytes) 000536 1755096506.004668 write 0x023b26 thru 0x02c631 (0x8b0c bytes) 000537 1755096506.009823 mapwrite 0x0237e6 thru 0x02f03f (0xb85a bytes) 000538 1755096506.042529 mapwrite 0x02f9fa thru 0x036a1d (0x7024 bytes) 000539 1755096506.068226 read 0x02c001 thru 0x034947 (0x8947 bytes) 000540 1755096506.068875 mapwrite 0x001e05 thru 0x00b506 (0x9702 bytes) 000543 1755096506.100137 mapread 0x018c45 thru 0x0260f0 (0xd4ac bytes) 000544 1755096506.100890 mapwrite 0x034016 thru 0x035a3e (0x1a29 bytes) 000547 1755096506.121771 read 0x034729 thru 0x03b0be (0x6996 bytes) 000549 1755096506.122874 mapwrite 0x02bb90 thru 0x0338ce (0x7d3f bytes) 000551 1755096506.150830 MIRROR_EXTEND to 5 mirrors 000552 1755096506.370041 mapwrite 0x030547 thru 0x03cea8 (0xc962 bytes) 000553 1755096506.475761 read 0x021b35 thru 0x027b32 (0x5ffe bytes) 000554 1755096506.481395 write 0x02ef44 thru 0x039d02 (0xadbf bytes) 000555 1755096506.482480 write 0x03178b thru 0x03562f (0x3ea5 bytes) 000556 1755096506.483374 read 0x021435 thru 0x030102 (0xecce bytes) 000558 1755096506.490761 mapread 0x006e2d thru 0x00eeaa (0x807e bytes) 000561 1755096506.492711 trunc from 0x03cea9 to 0x03b812 (0x1696 bytes) 000562 1755096506.509511 mapread 0x00375d thru 0x007bd0 (0x4474 bytes) 000563 1755096506.510435 read 0x019c82 thru 0x025ec5 (0xc244 bytes) 000564 1755096506.511008 trunc from 0x03b813 to 0x029d47 (0x11acb bytes) 000565 1755096506.522943 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000568 1755096506.619860 trunc from 0x029d48 to 0x026d11 (0x3036 bytes) 000569 1755096506.628681 trunc from 0x026d12 to 0x020c4f (0x60c2 bytes) 000570 1755096506.637040 mapread 0x017295 thru 0x01d529 (0x6295 bytes) 000571 1755096506.647666 read 0x01934f thru 0x01c96b (0x361d bytes) 000573 1755096506.649774 mapread 0x00688e thru 0x008704 (0x1e77 bytes) 000574 1755096506.655869 mapread 0x01f885 thru 0x020c4f (0x13cb bytes) 000575 1755096506.656509 write 0x0343d9 thru 0x037658 (0x3280 bytes) 000578 1755096506.658200 read 0x00b3b9 thru 0x00ed92 (0x39da bytes) 000580 1755096506.659279 trunc from 0x037659 to 0x00ddab (0x298ad bytes) 000583 1755096506.670621 mapread 0x00423f thru 0x00c3fc (0x81be bytes) 000584 1755096506.671490 write 0x031ad4 thru 0x03ffff (0xe52c bytes) 000590 1755096506.675096 mapread 0x00346a thru 0x005f60 (0x2af7 bytes) 000592 1755096506.676406 mapread 0x012763 thru 0x0175a5 (0x4e43 bytes) 000595 1755096506.683669 trunc from 0x040000 to 0x004bed (0x3b412 bytes) 000596 1755096506.694203 mapwrite 0x033d3a thru 0x03e88a (0xab51 bytes) 000597 1755096506.766996 trunc from 0x03e88b to 0x030cf6 (0xdb94 bytes) 000599 1755096506.780096 MIRROR_EXTEND to 5 mirrors 000600 1755096507.016769 MIRROR_RESYNC 5 mirrors 000601 1755096507.406237 MIRROR_EXTEND to 6 mirrors 000602 1755096507.636772 write 0x01b951 thru 0x028b11 (0xd1c1 bytes) 000604 1755096507.672909 write 0x031d0a thru 0x03c800 (0xaaf7 bytes) 000606 1755096507.677556 MIRROR_RESYNC 6 mirrors 000607 1755096508.024768 trunc from 0x03c801 to 0x02e008 (0xe7f8 bytes) 000610 1755096508.062346 mapwrite 0x00181a thru 0x00312d (0x1914 bytes) 000611 1755096508.100336 write 0x017499 thru 0x01ebc4 (0x772c bytes) 000613 1755096508.106251 mapwrite 0x01413e thru 0x01614a (0x200d bytes) 000614 1755096508.131031 trunc from 0x02e009 to 0x024738 (0x98d0 bytes) 000615 1755096508.141553 mapwrite 0x034382 thru 0x03eedc (0xab5b bytes) 000616 1755096508.187764 write 0x003fe6 thru 0x004017 (0x32 bytes) 000618 1755096508.192651 MIRROR_EXTEND to 7 mirrors 000619 1755096508.402984 write 0x00fa01 thru 0x01b142 (0xb742 bytes) 000620 1755096508.452226 read 0x008a9b thru 0x01587a (0xcde0 bytes) 000621 1755096508.458034 read 0x03b531 thru 0x03eedc (0x39ac bytes) 000622 1755096508.464777 mapwrite 0x00690a thru 0x0095bd (0x2cb4 bytes) 000624 1755096508.494916 read 0x00b3ea thru 0x016c77 (0xb88e bytes) 000625 1755096508.498437 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000627 1755096508.615231 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000628 1755096508.713844 trunc from 0x03eedd to 0x02d6b6 (0x11826 bytes) 000629 1755096508.734404 MIRROR_RESYNC 5 mirrors 000630 1755096509.161831 MIRROR_EXTEND to 6 mirrors 000631 1755096509.484328 read 0x024d36 thru 0x02d6b6 (0x8981 bytes) 000634 1755096509.489722 write 0x00baee thru 0x01ab4d (0xf060 bytes) 000636 1755096509.534904 mapwrite 0x0145c0 thru 0x01aeec (0x692d bytes) 000638 1755096509.566134 mapread 0x00c4e5 thru 0x00ff4a (0x3a66 bytes) 000639 1755096509.569632 MIRROR_EXTEND to 7 mirrors 000640 1755096509.785412 trunc from 0x02d6b7 to 0x011ce1 (0x1b9d5 bytes) 000641 1755096509.845872 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000644 1755096509.971668 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000645 1755096510.090820 write 0x02efab thru 0x03bbae (0xcc04 bytes) 000646 1755096510.092318 write 0x01e614 thru 0x023167 (0x4b54 bytes) 000647 1755096510.104440 read 0x02c46a thru 0x03993f (0xd4d6 bytes) 000648 1755096510.113543 MIRROR_RESYNC 5 mirrors 000649 1755096510.536047 mapread 0x011aa4 thru 0x01c27b (0xa7d8 bytes) 000651 1755096510.561176 MIRROR_RESYNC 5 mirrors 000653 1755096510.668140 write 0x023dcf thru 0x02bb4c (0x7d7e bytes) 000654 1755096510.714859 write 0x00405a thru 0x00c39e (0x8345 bytes) 000657 1755096510.722388 write 0x03f8a4 thru 0x03ffff (0x75c bytes) 000658 1755096510.723109 write 0x0002c5 thru 0x00f6c5 (0xf401 bytes) 000659 1755096510.727377 trunc from 0x040000 to 0x022344 (0x1dcbb bytes) 000660 1755096510.750027 read 0x0007c5 thru 0x0102ad (0xfae9 bytes) 000662 1755096510.751184 mapwrite 0x01ad93 thru 0x01c3b1 (0x161f bytes) 000663 1755096510.793087 mapread 0x00fb98 thru 0x01724b (0x76b4 bytes) 000665 1755096510.799650 read 0x01fa54 thru 0x022344 (0x28f1 bytes) 000666 1755096510.800219 write 0x01ea41 thru 0x026b8c (0x814c bytes) 000667 1755096510.801355 mapread 0x012839 thru 0x014dbc (0x2584 bytes) 000668 1755096510.802041 read 0x008d5d thru 0x00af21 (0x21c5 bytes) 000669 1755096510.802550 mapwrite 0x03c9e3 thru 0x03ffff (0x361d bytes) 000670 1755096510.855607 write 0x022604 thru 0x0273ce (0x4dcb bytes) 000673 1755096510.857020 trunc from 0x040000 to 0x031fbe (0xe041 bytes) 000674 1755096510.865609 read 0x02ccd2 thru 0x031fbe (0x52ed bytes) 000675 1755096510.869528 MIRROR_EXTEND to 6 mirrors 000676 1755096511.151583 mapwrite 0x005616 thru 0x00abc2 (0x55ad bytes) 000677 1755096511.241450 mapread 0x015d8f thru 0x023c3c (0xdeae bytes) 000678 1755096511.251589 mapwrite 0x036b2b thru 0x0382ea (0x17c0 bytes) 000680 1755096511.289816 read 0x032a93 thru 0x0382ea (0x5858 bytes) 000681 1755096511.290434 write 0x031f50 thru 0x03299e (0xa4f bytes) 000682 1755096511.295182 MIRROR_EXTEND to 7 mirrors 000683 1755096511.517409 MIRROR_RESYNC 7 mirrors 000684 1755096511.946081 mapread 0x01a2ab thru 0x01c7be (0x2514 bytes) 000688 1755096511.961045 MIRROR_RESYNC 7 mirrors 000690 1755096512.059590 read 0x0186ed thru 0x0251f8 (0xcb0c bytes) 000691 1755096512.062596 read 0x02171a thru 0x02bccb (0xa5b2 bytes) 000693 1755096512.068225 trunc from 0x0382eb to 0x037616 (0xcd4 bytes) 000694 1755096512.123707 mapread 0x00574d thru 0x00c8d7 (0x718b bytes) 000697 1755096512.147692 MIRROR_RESYNC 7 mirrors 000699 1755096512.610789 MIRROR_RESYNC 7 mirrors 000703 1755096512.728155 write 0x01c9ed thru 0x026a01 (0xa015 bytes) 000705 1755096512.779850 trunc from 0x037617 to 0x0260ac (0x1156a bytes) 000706 1755096512.808943 MIRROR_EXTEND to 8 mirrors 000708 1755096513.043399 mapwrite 0x01f750 thru 0x0270ca (0x797b bytes) 000710 1755096513.134641 write 0x01925e thru 0x021dd5 (0x8b78 bytes) 000711 1755096513.139231 trunc from 0x0270cb to 0x016c57 (0x10473 bytes) 000712 1755096513.149905 mapwrite 0x0352c5 thru 0x03f654 (0xa390 bytes) 000715 1755096513.206342 mapwrite 0x02ff91 thru 0x038d3b (0x8dab bytes) 000717 1755096513.235488 write 0x01098e thru 0x019f3e (0x95b1 bytes) 000718 1755096513.244220 read 0x018e60 thru 0x021b33 (0x8cd4 bytes) 000719 1755096513.250503 read 0x0118bf thru 0x01aaf7 (0x9239 bytes) 000722 1755096513.252108 trunc from 0x03f655 to 0x038ee8 (0x676c bytes) 000727 1755096513.266539 write 0x012cdf thru 0x01ac5a (0x7f7c bytes) 000728 1755096513.270675 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000731 1755096513.402900 mapwrite 0x027f08 thru 0x0351d8 (0xd2d1 bytes) 000732 1755096513.450355 write 0x000dd4 thru 0x0089d7 (0x7c04 bytes) 000733 1755096513.456394 mapwrite 0x00cb15 thru 0x01123f (0x472b bytes) 000734 1755096513.489360 read 0x00758e thru 0x010812 (0x9285 bytes) 000735 1755096513.489885 mapwrite 0x03da6c thru 0x03ffff (0x2594 bytes) 000736 1755096513.525517 write 0x015b5a thru 0x0253da (0xf881 bytes) 000737 1755096513.526943 read 0x00c29c thru 0x01438f (0x80f4 bytes) 000738 1755096513.527546 read 0x0307e3 thru 0x034fd2 (0x47f0 bytes) 000740 1755096513.528139 write 0x03905f thru 0x03ffff (0x6fa1 bytes) 000742 1755096513.529348 read 0x017318 thru 0x0256a9 (0xe392 bytes) 000743 1755096513.529902 read 0x02139f thru 0x02cf1a (0xbb7c bytes) 000744 1755096513.530486 mapwrite 0x01c9e3 thru 0x021bdc (0x51fa bytes) 000745 1755096513.556581 write 0x0327a2 thru 0x03e699 (0xbef8 bytes) 000750 1755096513.559550 write 0x01b834 thru 0x0259ec (0xa1b9 bytes) 000751 1755096513.560574 mapwrite 0x0059fb thru 0x00f017 (0x961d bytes) 000752 1755096513.589406 mapread 0x02abce thru 0x03023a (0x566d bytes) 000753 1755096513.590263 trunc from 0x040000 to 0x01ec7b (0x21384 bytes) 000755 1755096513.599842 read 0x019892 thru 0x01bd60 (0x24cf bytes) 000756 1755096513.600400 write 0x001dad thru 0x00faca (0xdd1e bytes) 000757 1755096513.601408 mapwrite 0x03bc5e thru 0x03ffff (0x43a2 bytes) 000759 1755096513.647700 MIRROR_RESYNC 7 mirrors 000760 1755096514.147753 MIRROR_RESYNC 7 mirrors 000762 1755096514.253436 read 0x0169a0 thru 0x01d517 (0x6b78 bytes) 000766 1755096514.258215 mapread 0x00fc99 thru 0x01cd0c (0xd074 bytes) 000768 1755096514.267486 read 0x03bf91 thru 0x03ffff (0x406f bytes) 000770 1755096514.274729 mapwrite 0x03de00 thru 0x03ffff (0x2200 bytes) 000772 1755096514.358590 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000775 1755096514.468673 write 0x021670 thru 0x02553c (0x3ecd bytes) 000776 1755096514.475389 mapwrite 0x01f955 thru 0x02916a (0x9816 bytes) 000779 1755096514.519199 mapread 0x0322e7 thru 0x032525 (0x23f bytes) 000780 1755096514.526702 mapread 0x028b30 thru 0x03660a (0xdadb bytes) 000781 1755096514.527829 mapread 0x021101 thru 0x029610 (0x8510 bytes) 000784 1755096514.532092 MIRROR_RESYNC 6 mirrors 000785 1755096514.909806 mapwrite 0x032fca thru 0x03f7fd (0xc834 bytes) 000786 1755096515.017801 read 0x00cdce thru 0x01a30a (0xd53d bytes) 000788 1755096515.024169 write 0x01d048 thru 0x022503 (0x54bc bytes) 000792 1755096515.032843 MIRROR_EXTEND to 7 mirrors 000793 1755096515.281894 trunc from 0x040000 to 0x02042c (0x1fbd3 bytes) 000796 1755096515.354001 write 0x030bbf thru 0x03abac (0x9fee bytes) 000797 1755096515.355375 read 0x0135ea thru 0x021bb1 (0xe5c8 bytes) 000798 1755096515.363016 read 0x0194af thru 0x020905 (0x7457 bytes) 000801 1755096515.364612 mapread 0x00674d thru 0x0158c2 (0xf176 bytes) 000803 1755096515.372858 trunc from 0x03abad to 0x034822 (0x638a bytes) 000804 1755096515.388053 MIRROR_EXTEND to 8 mirrors 000807 1755096515.629796 write 0x020631 thru 0x02d3ed (0xcdbd bytes) 000808 1755096515.679154 MIRROR_RESYNC 8 mirrors 000809 1755096516.248086 mapread 0x01df82 thru 0x02ca04 (0xea83 bytes) 000810 1755096516.263410 trunc from 0x034823 to 0x0013f2 (0x33430 bytes) 000812 1755096516.303113 mapwrite 0x02ed80 thru 0x034fca (0x624b bytes) 000813 1755096516.352702 mapread 0x005d40 thru 0x011d86 (0xc047 bytes) 000814 1755096516.365155 mapwrite 0x02e1c6 thru 0x0317c0 (0x35fb bytes) 000816 1755096516.385558 mapwrite 0x00c5c8 thru 0x015cd1 (0x970a bytes) 000817 1755096516.429441 MIRROR_EXTEND to 9 mirrors 000818 1755096516.668819 read 0x00b002 thru 0x015720 (0xa71f bytes) 000819 1755096516.679224 mapwrite 0x00f84d thru 0x0165fa (0x6dae bytes) 000821 1755096516.781998 read 0x017a1c thru 0x021d27 (0xa30c bytes) 000824 1755096516.791806 MIRROR_EXTEND to 10 mirrors 000825 1755096517.035019 write 0x038d36 thru 0x03ffff (0x72ca bytes) 000827 1755096517.079078 read 0x002062 thru 0x010227 (0xe1c6 bytes) 000829 1755096517.085232 mapwrite 0x00bcaa thru 0x016929 (0xac80 bytes) 000830 1755096517.127017 trunc from 0x040000 to 0x0067d9 (0x39826 bytes) 000831 1755096517.151525 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000832 1755096517.259807 read 0x0042f2 thru 0x0048ab (0x5ba bytes) 000833 1755096517.263563 trunc from 0x0067da to 0x029d80 (0x235a7 bytes) 000834 1755096517.273416 write 0x001ed8 thru 0x00dd1a (0xbe43 bytes) 000836 1755096517.284199 write 0x02624f thru 0x031b28 (0xb8da bytes) 000841 1755096517.294139 read 0x019cd0 thru 0x028aa7 (0xedd8 bytes) 000843 1755096517.305843 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000845 1755096517.428032 read 0x015d09 thru 0x021bb5 (0xbead bytes) 000846 1755096517.434162 read 0x0207c7 thru 0x02f359 (0xeb93 bytes) 000848 1755096517.441167 read 0x0209a0 thru 0x02a231 (0x9892 bytes) 000849 1755096517.445864 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000850 1755096517.564393 read 0x014fd0 thru 0x01502d (0x5e bytes) 000851 1755096517.576042 MIRROR_RESYNC 7 mirrors 000853 1755096518.086008 write 0x01b25a thru 0x02797a (0xc721 bytes) 000855 1755096518.132190 write 0x01aff6 thru 0x025597 (0xa5a2 bytes) 000856 1755096518.139190 trunc from 0x031b29 to 0x00dfa8 (0x23b80 bytes) 000858 1755096518.175542 MIRROR_EXTEND to 8 mirrors 000860 1755096518.463623 read 0x003965 thru 0x00dfa8 (0xa644 bytes) 000861 1755096518.473936 mapread 0x0060fc thru 0x008509 (0x240e bytes) 000862 1755096518.482904 write 0x013008 thru 0x01e716 (0xb70f bytes) 000863 1755096518.520034 read 0x0067c7 thru 0x009acc (0x3306 bytes) 000864 1755096518.527427 trunc from 0x01e717 to 0x02922a (0xab14 bytes) 000866 1755096518.546089 mapread 0x00c713 thru 0x018da9 (0xc697 bytes) 000869 1755096518.547962 mapwrite 0x02b963 thru 0x02fbb5 (0x4253 bytes) 000870 1755096518.606178 trunc from 0x02fbb6 to 0x02e235 (0x1980 bytes) 000871 1755096518.625653 trunc from 0x02e236 to 0x022d77 (0xb4be bytes) 000872 1755096518.636910 read 0x01c941 thru 0x022d77 (0x6437 bytes) 000873 1755096518.637598 trunc from 0x022d78 to 0x02867a (0x5903 bytes) 000876 1755096518.651234 mapwrite 0x025926 thru 0x026ff4 (0x16cf bytes) 000878 1755096518.704943 trunc from 0x02867b to 0x009c9b (0x1e9df bytes) 000879 1755096518.718408 read 0x0056a0 thru 0x00713d (0x1a9e bytes) 000880 1755096518.719015 read 0x0047df thru 0x009c9b (0x54bd bytes) 000883 1755096518.726063 MIRROR_RESYNC 8 mirrors 000884 1755096519.302896 trunc from 0x009c9c to 0x00b1e2 (0x1547 bytes) 000885 1755096519.348735 mapwrite 0x017326 thru 0x01d5f9 (0x62d4 bytes) 000888 1755096519.401484 trunc from 0x01d5fa to 0x0198bc (0x3d3d bytes) 000889 1755096519.409426 read 0x010114 thru 0x012d02 (0x2bef bytes) 000890 1755096519.413134 trunc from 0x0198bd to 0x01e41e (0x4b62 bytes) 000891 1755096519.424390 MIRROR_RESYNC 8 mirrors 000892 1755096519.942157 trunc from 0x01e41f to 0x03a86c (0x1c44e bytes) 000893 1755096519.988698 MIRROR_RESYNC 8 mirrors 000894 1755096520.484279 write 0x01528e thru 0x01b78b (0x64fe bytes) 000896 1755096520.521499 MIRROR_EXTEND to 9 mirrors 000897 1755096520.785859 mapread 0x0047c9 thru 0x008471 (0x3ca9 bytes) 000898 1755096520.804182 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000899 1755096520.964792 MIRROR_RESYNC 8 mirrors 000901 1755096521.399190 write 0x028d8f thru 0x031af4 (0x8d66 bytes) 000902 1755096521.441945 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000903 1755096521.543166 mapread 0x030592 thru 0x03a86c (0xa2db bytes) 000904 1755096521.556634 MIRROR_EXTEND to 8 mirrors 000905 1755096521.752444 mapread 0x00070b thru 0x003025 (0x291b bytes) 000909 1755096521.777642 read 0x0008ff thru 0x00f691 (0xed93 bytes) 000910 1755096521.782269 MIRROR_EXTEND to 9 mirrors 000911 1755096521.980174 read 0x0305bd thru 0x03742f (0x6e73 bytes) 000912 1755096521.990536 MIRROR_EXTEND to 10 mirrors 000915 1755096522.238873 mapread 0x02d4c3 thru 0x03822e (0xad6c bytes) 000916 1755096522.252934 trunc from 0x03a86d to 0x005917 (0x34f55 bytes) 000918 1755096522.306846 read 0x002c12 thru 0x005917 (0x2d06 bytes) 000922 1755096522.311895 read 0x004816 thru 0x005917 (0x1102 bytes) 000924 1755096522.312887 mapread 0x00356f thru 0x005917 (0x23a9 bytes) 000926 1755096522.314756 write 0x01f484 thru 0x02bf3d (0xcaba bytes) 000927 1755096522.316956 mapwrite 0x029710 thru 0x02d409 (0x3cfa bytes) 000929 1755096522.358638 trunc from 0x02d40a to 0x00a1e5 (0x23224 bytes) 000930 1755096522.367164 mapwrite 0x03cf53 thru 0x03ffff (0x30ad bytes) 000932 1755096522.402585 mapread 0x0049b4 thru 0x009baf (0x51fc bytes) 000933 1755096522.403346 read 0x038b9b thru 0x03cbe2 (0x4048 bytes) 000934 1755096522.403899 write 0x03b387 thru 0x03ffff (0x4c79 bytes) 000935 1755096522.404656 write 0x02b113 thru 0x0315ca (0x64b8 bytes) 000936 1755096522.412273 MIRROR_RESYNC 10 mirrors 000937 1755096522.943345 trunc from 0x040000 to 0x001f5c (0x3e0a3 bytes) 000938 1755096522.990567 MIRROR_EXTEND to 11 mirrors 000939 1755096523.174746 MIRROR_EXTEND to 12 mirrors 000940 1755096523.336521 mapread 0x00093f thru 0x001f5c (0x161e bytes) 000941 1755096523.341419 write 0x00cd8f thru 0x00ef02 (0x2174 bytes) 000942 1755096523.370366 mapwrite 0x01b62d thru 0x02a59a (0xef6e bytes) 000945 1755096523.408499 mapread 0x00411a thru 0x0083fd (0x42e4 bytes) 000946 1755096523.412518 trunc from 0x02a59b to 0x00e058 (0x1c542 bytes) 000947 1755096523.418487 write 0x020b99 thru 0x026285 (0x56ed bytes) 000949 1755096523.421725 MIRROR_EXTEND to 13 mirrors 000951 1755096523.599186 mapwrite 0x020a17 thru 0x027a5f (0x7049 bytes) 000952 1755096523.653918 MIRROR_EXTEND to 14 mirrors 000953 1755096523.832216 mapread 0x01d768 thru 0x027a5f (0xa2f8 bytes) 000955 1755096523.843151 mapwrite 0x00aa36 thru 0x015786 (0xad51 bytes) 000956 1755096523.926445 read 0x00bace thru 0x0196e5 (0xdc18 bytes) 000957 1755096523.927039 read 0x00e030 thru 0x01033f (0x2310 bytes) 000958 1755096523.927530 read 0x007778 thru 0x016cae (0xf537 bytes) 000959 1755096523.927987 write 0x0103e7 thru 0x01f9d1 (0xf5eb bytes) 000962 1755096523.933720 trunc from 0x027a60 to 0x00602b (0x21a34 bytes) 000964 1755096523.948891 write 0x00b2db thru 0x0195f0 (0xe316 bytes) 000965 1755096523.950148 read 0x00dc93 thru 0x017c7d (0x9feb bytes) 000966 1755096523.950606 write 0x011307 thru 0x01bf32 (0xac2c bytes) 000967 1755096523.951493 mapread 0x01ab98 thru 0x01bf32 (0x139b bytes) 000968 1755096523.951949 read 0x013c22 thru 0x01bf32 (0x8311 bytes) 000970 1755096523.952844 mapwrite 0x02c51e thru 0x03b6d8 (0xf1bb bytes) 000971 1755096523.987146 mapwrite 0x007244 thru 0x00eca8 (0x7a65 bytes) 000973 1755096524.012513 mapwrite 0x0274cb thru 0x0304f3 (0x9029 bytes) 000974 1755096524.037890 mapread 0x017285 thru 0x01bb64 (0x48e0 bytes) 000975 1755096524.041447 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000976 1755096524.159003 mapwrite 0x020570 thru 0x02ba05 (0xb496 bytes) 000977 1755096524.191691 trunc from 0x03b6d9 to 0x021b6f (0x19b69 bytes) 000979 1755096524.199714 write 0x03ed37 thru 0x03ffff (0x12c9 bytes) 000980 1755096524.200682 read 0x01ca5a thru 0x022d7b (0x6322 bytes) 000981 1755096524.203466 trunc from 0x040000 to 0x036248 (0x9db7 bytes) 000982 1755096524.210656 trunc from 0x036249 to 0x0383d1 (0x2189 bytes) 000984 1755096524.219433 trunc from 0x0383d2 to 0x03201e (0x63b3 bytes) 000985 1755096524.226857 read 0x0265c4 thru 0x02e3cd (0x7e0a bytes) 000986 1755096524.234231 MIRROR_RESYNC 13 mirrors 000989 1755096524.732880 write 0x014a13 thru 0x01fad4 (0xb0c2 bytes) 000991 1755096524.758397 trunc from 0x03201f to 0x03bfd1 (0x9fb3 bytes) 000992 1755096524.772262 mapwrite 0x02f1b1 thru 0x038156 (0x8fa6 bytes) 000993 1755096524.800491 mapwrite 0x03cea2 thru 0x03ffff (0x315e bytes) 000994 1755096524.826558 mapread 0x00a82b thru 0x019d3b (0xf511 bytes) 000995 1755096524.834224 read 0x009f88 thru 0x00c228 (0x22a1 bytes) 000997 1755096524.837813 MIRROR_SPLIT mirror 5 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3c0:0x0]: Invalid argument (22) 000998 1755096524.922951 mapwrite 0x011b1d thru 0x0207b3 (0xec97 bytes) 000999 1755096524.950197 trunc from 0x040000 to 0x0330e6 (0xcf19 bytes) 001000 1755096524.956309 write 0x0219c0 thru 0x0224f6 (0xb37 bytes) All operations completed A-OK! PASS 70a (33s) == sanity-flr test 71: check mirror extend parent fid ==== 10:48:47 (1755096527) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 10:48:48 (1755096528) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2837888 .. Extending file size to 3492896 .. failed Extending file size to 6070432 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6073984 .. Extending file size to 7630432 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6452064 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7742048 .. Extending file size to 7822688 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8405504 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7887264 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8793344 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8973792 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9118592 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9127712 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9283616 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 56324 56326 56327 56329 56331 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) 192.168.202.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:49:56 (1755096596) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5031552 .. Extending file size to 2972928 .. done Extending file size to 5112512 .. Extending file size to 7288160 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7988640 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8630976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6932864 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8627648 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8941984 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8878976 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9181248 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8897856 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9263072 .. failed Extending file size to 8917568 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8922944 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 59213 59214 59216 59218 59220 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) 192.168.202.105@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:51:02 (1755096662) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 4608 2203776 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0449647 s, 46.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111939 s, 18.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:51:09 (1755096669) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0367594 s, 57.1 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 10:51:12 (1755096672) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x409:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x587:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x587:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x588:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x58a:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:51:15 (1755096675) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0405281 s, 51.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x589:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:51:18 (1755096678) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0632673 s, 33.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58e:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:51:21 (1755096681) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0501022 s, 41.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x590:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:51:47 (1755096707) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562334 s, 18.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x592:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x591:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x592:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x591:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58e:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:52:35 (1755096755) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x590:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0639115 s, 32.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x590:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x593:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x590:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x593:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:53:21 (1755096801) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg205-server mds-ost sync done. Creating new pool test_204f oleg205-server: Pool lustre.test_204f created Adding targets to pool oleg205-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg205-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0416627 s, 50.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x591:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x594:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x591:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x594:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg205-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg205-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg205-server: Pool lustre.test_204f destroyed PASS 204f (57s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:54:18 (1755096858) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:54:21 (1755096861) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:54:24 (1755096864) oleg205-server: Pool lustre.test_206 created Adding targets to pool oleg205-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg205-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg205-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg205-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg205-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:54:40 (1755096880) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283307 s, 37.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122506 s, 85.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x44f6740e Mirror 3: 0x44f6740e PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:54:43 (1755096883) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:54:45 (1755096885) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:54:46 (1755096886) Starting client: oleg205-client.virtnet: -o user_xattr,flock 192.168.202.105@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0382402 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb40000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ae0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ae0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ae0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2620000 Read this (4 bytes): BBBB 192.168.202.105@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg205-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:55:04 (1755096904) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00515291 s, 1.6 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:55:07 (1755096907) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162853 s, 64.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x42e lmm_fid: [0x200000404:0x42e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x42e lmm_fid: [0x200000404:0x42e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x59e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834990 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-08-13 10:55:08.000000000 -0400 Modify: 2025-08-13 10:55:08.000000000 -0400 Change: 2025-08-13 10:55:08.000000000 -0400 Birth: 2025-08-13 10:55:08.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:55:10 (1755096910) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:55:11 (1755096911) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.145379 s, 282 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0150513 s, 272 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 13 10:55 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x430:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 13 10:55 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 921 sec ============ 10:55:14 (1755096914) === sanity-flr: start cleanup 10:55:15 (1755096915) === === sanity-flr: finish cleanup 10:55:17 (1755096917) ===