-----============= acceptance-small: sanity-flr ============----- Tue Aug 12 18:35:16 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg425-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:35:23 (1755038123) === oleg425-client.virtnet: executing check_config_client /mnt/lustre oleg425-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg425-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99ba208bd800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99ba208bd800.idle_timeout=debug disable quota as required oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 18:35:32 (1755038132) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7542] == sanity-flr test 0a: lfs mirror create with -N option == 18:35:32 (1755038132) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:35:37 (1755038137) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:35:38 (1755038138) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:35:39 (1755038139) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:35:45 (1755038145) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:35:46 (1755038146) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 18:35:47 (1755038147) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff99ba208bd800.stats=clear osc.lustre-OST0001-osc-ffff99ba208bd800.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:35:49 (1755038149) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:35:52 (1755038152) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:35:55 (1755038155) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:35:56 (1755038156) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (2s) == sanity-flr test 2: create components from existing files ========================================================== 18:35:58 (1755038158) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:36:01 (1755038161) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (0s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:36:02 (1755038162) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:36:04 (1755038164) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:36:07 (1755038167) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:36:09 (1755038169) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0608611 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349254 s, 30.0 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:36:11 (1755038171) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0216698 s, 96.8 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:36:14 (1755038174) Stopping /mnt/lustre-ost1 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ba208bd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ba208bd800.ost_server_uuid in DISCONN state after 3 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg425-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99ba208bd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ba208bd800.ost_server_uuid in DISCONN state after 2 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg425-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:36:41 (1755038201) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0874446 s, 60.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ba208bd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ba208bd800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg425-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (12s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:36:53 (1755038213) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ba208bd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ba208bd800.ost_server_uuid in DISCONN state after 4 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg425-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99ba208bd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ba208bd800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg425-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (25s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:37:18 (1755038238) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.888217 s, 59.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 12 18:37 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.296s user 0m0.001s sys 0m0.148s Stopping /mnt/lustre-ost1 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff99ba208bd800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99ba208bd800.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.259s user 0m0.000s sys 0m0.146s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg425-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg425-server oleg425-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff99ba208bd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ba208bd800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.239s user 0m0.001s sys 0m0.136s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg425-server: oleg425-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg425-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg425-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99ba208bd800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99ba208bd800.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:37:46 (1755038266) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:37:47 (1755038267) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:37:48 (1755038268) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 18:37:49 (1755038269) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 18:37:51 (1755038271) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.129045 s, 97.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:37:54 (1755038274) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00978251 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844702 s, 124 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:37:56 (1755038276) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:37:59 (1755038279) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00044387 s, 2.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000467541 s, 2.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000440803 s, 2.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000623862 s, 1.6 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 18:38:02 (1755038282) create test files with size 15743267 .. c5d994bc01b8ae2cf361c191047436ba - e7a7270a26b8b16079b3635b39f26b29 - fa950bfe076ae7fc18731dac370c8adc - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg425-client.virtnet /mnt/lustre (opts:) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 18:38:07 (1755038287) 4 mirror IDs: 1 2 3 4 mirror_io: 415: hit by error injection mirror_io: 415: hit by error injection PASS 38 (3s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:38:10 (1755038290) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:38:12 (1755038292) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292057 s, 35.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x47:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x48:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x43:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00907647 s, 116 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x44:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 18:38:15 (1755038295) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0319497 s, 65.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0899279 s, 46.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0461802 s, 45.4 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (2s) == sanity-flr test 42: lfs mirror verify ================= 18:38:18 (1755038298) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (0s) == sanity-flr test 43a: mirror pick on write ============= 18:38:18 (1755038298) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:38:19 (1755038299) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00958276 s, 109 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 18:38:22 (1755038302) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:38:23 (1755038303) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4c:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x50:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:38:25 (1755038305) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0702704 s, 149 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg425-client.virtnet /mnt/lustre (opts:) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (23s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:38:48 (1755038328) llite.lustre-ffff99ba066b1000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0730239 s, 144 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg425-client.virtnet /mnt/lustre (opts:) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff99ba205ca800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:38:52 (1755038332) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (12s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:39:04 (1755038344) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (13s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:39:17 (1755038357) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 18:39:18 (1755038358) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0594262 s, 52.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0501376 s, 62.7 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 8b815d08cd1853092d68595de99fd1ce - mirror 1 checksum: bd03a60cc0b426d6dd064922285320d8 - mirror 2 checksum: 8b815d08cd1853092d68595de99fd1ce - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Aug 12 18:39:20 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Tue Aug 12 18:39:20 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8b815d08cd1853092d68595de99fd1ce - mirror 1 checksum: bd03a60cc0b426d6dd064922285320d8 - mirror 2 checksum: bd03a60cc0b426d6dd064922285320d8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Aug 12 18:39:20 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x62:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Tue Aug 12 18:39:20 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x63:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:39:22 (1755038362) Failing mds1 on oleg425-server Stopping /mnt/lustre-mds1 (opts:) on oleg425-server 18:39:24 (1755038364) shut down facet: mds1 facet_host: oleg425-server facet_failover_host: oleg425-server Failover mds1 to oleg425-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 18:39:39 (1755038379) targets are mounted 18:39:39 (1755038379) facet_failover done oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144892 s, 72.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } PASS 50A (25s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:39:47 (1755038387) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:39:48 (1755038388) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:39:49 (1755038389) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00267879 s, 6.1 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:39:51 (1755038391) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:39:52 (1755038392) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:39:53 (1755038393) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:40:02 (1755038402) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:40:21 (1755038421) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:40:40 (1755038440) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:40:58 (1755038458) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 18:41:01 (1755038461) mirror create pid 52266 mirror split pid 52267 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 52266 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4953: 52267 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 18:42:04 (1755038524) Seed set to 8525 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x805e 000004 1755038525.602727 trunc from 00000000 to 0x00805d (0x805e bytes) 000005 1755038525.611944 mapread 0x0072ba thru 0x00805d (0xda4 bytes) 000006 1755038525.619237 mapread 0x0054a6 thru 0x00805d (0x2bb8 bytes) 000007 1755038525.624969 mapwrite 0x02679d thru 0x0276ed (0xf51 bytes) 000008 1755038525.696631 read 0x011a41 thru 0x01dfb8 (0xc578 bytes) 000010 1755038525.703310 mapread 0x012cd6 thru 0x01b175 (0x84a0 bytes) 000011 1755038525.704359 write 0x001e76 thru 0x009b54 (0x7cdf bytes) 000012 1755038525.708422 mapwrite 0x01fceb thru 0x027ec8 (0x81de bytes) 000013 1755038525.749974 read 0x001b1d thru 0x010952 (0xee36 bytes) 000015 1755038525.756318 mapwrite 0x026658 thru 0x035154 (0xeafd bytes) 000016 1755038525.803502 mapread 0x013a84 thru 0x020d60 (0xd2dd bytes) 000017 1755038525.804612 write 0x00de2a thru 0x0125ea (0x47c1 bytes) 000019 1755038525.806257 mapread 0x014af2 thru 0x01507d (0x58c bytes) truncating to largest ever: 0x2d660 000020 1755038525.807223 trunc from 0x035155 to 0x02d65f (0x7af5 bytes) 000022 1755038525.820004 MIRROR_EXTEND to 2 mirrors 000023 1755038526.033167 MIRROR_RESYNC 2 mirrors 000024 1755038526.105872 write 0x020a0f thru 0x02eda6 (0xe398 bytes) 000025 1755038526.144302 read 0x019f03 thru 0x01e10f (0x420d bytes) 000026 1755038526.149918 mapwrite 0x037629 thru 0x03ffff (0x89d7 bytes) 000030 1755038526.214718 trunc from 0x040000 to 0x0092c5 (0x36d3a bytes) 000034 1755038526.226320 trunc from 0x0092c6 to 0x00b0ec (0x1e27 bytes) 000035 1755038526.235161 write 0x01c869 thru 0x027768 (0xaf00 bytes) 000036 1755038526.236816 mapwrite 0x01e63e thru 0x0289eb (0xa3ae bytes) 000037 1755038526.280008 mapwrite 0x026df7 thru 0x02d6b7 (0x68c1 bytes) 000039 1755038526.326532 read 0x01c6f1 thru 0x01d63e (0xf4e bytes) truncating to largest ever: 0x2ecaa 000040 1755038526.327276 trunc from 0x02d6b8 to 0x02eca9 (0x15f2 bytes) 000041 1755038526.336503 mapwrite 0x00c7a5 thru 0x0173c3 (0xac1f bytes) 000043 1755038526.374257 mapread 0x01acb9 thru 0x0294b0 (0xe7f8 bytes) 000044 1755038526.375394 write 0x02f86b thru 0x0307a1 (0xf37 bytes) 000046 1755038526.377126 trunc from 0x0307a2 to 0x00b6ee (0x250b3 bytes) 000047 1755038526.387158 write 0x022d21 thru 0x029cb0 (0x6f90 bytes) 000048 1755038526.388628 mapwrite 0x0272d9 thru 0x033e46 (0xcb6e bytes) 000049 1755038526.446351 write 0x0048cf thru 0x00ec0f (0xa341 bytes) 000050 1755038526.450399 write 0x0101ec thru 0x0119d0 (0x17e5 bytes) 000053 1755038526.466931 write 0x02b825 thru 0x039df9 (0xe5d5 bytes) 000054 1755038526.468498 mapwrite 0x02b0ef thru 0x038b2b (0xda3d bytes) truncating to largest ever: 0x36a6c 000055 1755038526.512054 trunc from 0x039dfa to 0x036a6b (0x338e bytes) truncating to largest ever: 0x37e2f 000058 1755038526.523130 trunc from 0x036a6c to 0x037e2e (0x13c3 bytes) 000059 1755038526.532175 mapread 0x02b595 thru 0x0350d9 (0x9b45 bytes) 000063 1755038526.534980 mapwrite 0x006e08 thru 0x010813 (0x9a0c bytes) 000064 1755038526.571729 read 0x0114d9 thru 0x01f6ee (0xe216 bytes) 000065 1755038526.577780 read 0x000e35 thru 0x0102e0 (0xf4ac bytes) 000066 1755038526.581292 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000068 1755038526.656733 read 0x01880b thru 0x01abab (0x23a1 bytes) 000072 1755038526.662178 mapwrite 0x0002d6 thru 0x007a7f (0x77aa bytes) 000076 1755038526.705612 mapread 0x013777 thru 0x01d424 (0x9cae bytes) 000077 1755038526.715714 mapwrite 0x027c70 thru 0x02dbe3 (0x5f74 bytes) 000078 1755038526.746189 mapwrite 0x01914f thru 0x0198bb (0x76d bytes) 000079 1755038526.770749 read 0x020927 thru 0x024aac (0x4186 bytes) 000083 1755038526.773128 mapwrite 0x004094 thru 0x013ccb (0xfc38 bytes) 000084 1755038526.807304 MIRROR_EXTEND to 2 mirrors 000085 1755038527.028487 mapwrite 0x01c1b4 thru 0x024037 (0x7e84 bytes) 000086 1755038527.126901 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000090 1755038527.201856 trunc from 0x037e2f to 0x020521 (0x1790d bytes) truncating to largest ever: 0x3d7b6 000092 1755038527.223752 trunc from 0x020522 to 0x03d7b5 (0x1d294 bytes) 000093 1755038527.233743 write 0x037f95 thru 0x0385c4 (0x630 bytes) truncating to largest ever: 0x3dd05 000096 1755038527.241115 trunc from 0x03d7b6 to 0x03dd04 (0x54f bytes) 000097 1755038527.249846 read 0x0308c5 thru 0x03ad38 (0xa474 bytes) 000099 1755038527.256088 trunc from 0x03dd05 to 0x033cd0 (0xa034 bytes) 000100 1755038527.265745 mapread 0x02ae28 thru 0x033cd0 (0x8ea9 bytes) 000101 1755038527.272243 read 0x00961a thru 0x017e3f (0xe826 bytes) 000103 1755038527.279101 mapread 0x005e05 thru 0x012751 (0xc94d bytes) 000104 1755038527.280334 trunc from 0x033cd1 to 0x014673 (0x1f65d bytes) 000110 1755038527.292980 mapread 0x002bce thru 0x0030af (0x4e2 bytes) 000112 1755038527.294458 mapread 0x00a998 thru 0x010bc3 (0x622c bytes) 000113 1755038527.295440 trunc from 0x014674 to 0x022ae9 (0xe476 bytes) 000116 1755038527.305232 write 0x03b60c thru 0x03c0d3 (0xac8 bytes) 000117 1755038527.306371 read 0x0269c6 thru 0x02a9aa (0x3fe5 bytes) 000118 1755038527.313049 read 0x00916d thru 0x009c14 (0xaa8 bytes) 000121 1755038527.314796 trunc from 0x03c0d4 to 0x02e647 (0xda8c bytes) 000122 1755038527.322443 mapread 0x0243a5 thru 0x024c52 (0x8ae bytes) 000125 1755038527.324147 write 0x0323a3 thru 0x03b3f6 (0x9054 bytes) 000126 1755038527.325295 write 0x035af9 thru 0x03ffff (0xa507 bytes) 000127 1755038527.326572 read 0x00ac1a thru 0x0107b7 (0x5b9e bytes) 000128 1755038527.329533 MIRROR_EXTEND to 2 mirrors 000129 1755038527.468533 trunc from 0x040000 to 0x0141c8 (0x2be37 bytes) 000130 1755038527.498993 write 0x02811a thru 0x032a64 (0xa94b bytes) 000131 1755038527.500196 trunc from 0x032a65 to 0x0072d4 (0x2b790 bytes) 000132 1755038527.507114 mapread 0x001224 thru 0x0072d4 (0x60b1 bytes) 000133 1755038527.513094 mapread 0x0070ea thru 0x0072d4 (0x1eb bytes) 000137 1755038527.514446 mapwrite 0x02592e thru 0x02955b (0x3c2e bytes) 000138 1755038527.539869 mapwrite 0x012a94 thru 0x0220a8 (0xf615 bytes) 000140 1755038527.566331 read 0x0194e2 thru 0x01c035 (0x2b54 bytes) 000142 1755038527.569020 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000143 1755038527.608427 read 0x0285b1 thru 0x02955b (0xfab bytes) 000145 1755038527.610620 mapwrite 0x015d46 thru 0x019e09 (0x40c4 bytes) 000146 1755038527.632960 MIRROR_EXTEND to 2 mirrors 000148 1755038527.761475 trunc from 0x02955c to 0x0303f0 (0x6e95 bytes) 000149 1755038527.797861 mapread 0x007b9e thru 0x0165ed (0xea50 bytes) 000153 1755038527.810365 write 0x008755 thru 0x0112f5 (0x8ba1 bytes) 000154 1755038527.811617 trunc from 0x0303f1 to 0x00a2a9 (0x26147 bytes) 000155 1755038527.820686 MIRROR_RESYNC 2 mirrors 000156 1755038527.958799 mapread 0x0038e2 thru 0x003ba3 (0x2c2 bytes) 000157 1755038527.961948 mapread 0x0063bb thru 0x00a2a9 (0x3eef bytes) 000158 1755038527.969235 MIRROR_RESYNC 2 mirrors 000159 1755038528.031238 read 0x00205d thru 0x00a2a9 (0x824d bytes) 000160 1755038528.031731 read 0x0087dd thru 0x00a2a9 (0x1acd bytes) 000162 1755038528.032511 mapread 0x007aa2 thru 0x00929b (0x17fa bytes) 000164 1755038528.033705 mapwrite 0x0140e8 thru 0x01ab2c (0x6a45 bytes) 000165 1755038528.103256 mapwrite 0x032e1a thru 0x0343eb (0x15d2 bytes) 000166 1755038528.131112 trunc from 0x0343ec to 0x02e2b7 (0x6134 bytes) 000169 1755038528.139438 mapwrite 0x014a7f thru 0x01ab55 (0x60d7 bytes) 000171 1755038528.167119 write 0x01541f thru 0x017401 (0x1fe3 bytes) 000172 1755038528.168729 mapread 0x019b28 thru 0x01ee7d (0x5356 bytes) 000174 1755038528.170951 trunc from 0x02e2b8 to 0x032e90 (0x4bd9 bytes) 000175 1755038528.180219 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000178 1755038528.254307 write 0x006152 thru 0x01170f (0xb5be bytes) 000179 1755038528.260414 write 0x03866e thru 0x03ffff (0x7992 bytes) 000181 1755038528.262840 mapread 0x01876a thru 0x01b136 (0x29cd bytes) 000183 1755038528.271229 MIRROR_EXTEND to 2 mirrors 000184 1755038528.524117 write 0x01bfe9 thru 0x02aa6a (0xea82 bytes) 000185 1755038528.573334 mapwrite 0x001a9b thru 0x00e9d6 (0xcf3c bytes) 000186 1755038528.614323 mapwrite 0x03c14d thru 0x03ffff (0x3eb3 bytes) 000188 1755038528.639454 trunc from 0x040000 to 0x0263be (0x19c41 bytes) 000190 1755038528.652382 mapread 0x0079d5 thru 0x00e2b4 (0x68e0 bytes) 000191 1755038528.656373 MIRROR_EXTEND to 3 mirrors 000192 1755038528.860275 mapread 0x013b06 thru 0x023705 (0xfc00 bytes) 000194 1755038528.874769 write 0x009c95 thru 0x0104c4 (0x6830 bytes) 000197 1755038528.920523 read 0x01fcd4 thru 0x0263be (0x66eb bytes) 000198 1755038528.926017 write 0x019e29 thru 0x01d71a (0x38f2 bytes) 000199 1755038528.932730 read 0x0162a5 thru 0x0184be (0x221a bytes) 000200 1755038528.933355 trunc from 0x0263bf to 0x00bb8a (0x1a834 bytes) 000203 1755038528.956188 write 0x021c85 thru 0x026528 (0x48a4 bytes) 000204 1755038528.957538 mapwrite 0x01a99e thru 0x0241b8 (0x981b bytes) 000205 1755038528.998156 mapread 0x014e25 thru 0x0239e0 (0xebbc bytes) 000207 1755038528.999957 read 0x00d6e2 thru 0x013467 (0x5d86 bytes) 000208 1755038529.000750 write 0x024433 thru 0x02d25a (0x8e28 bytes) 000210 1755038529.002850 mapread 0x026f8e thru 0x02d25a (0x62cd bytes) 000211 1755038529.003909 trunc from 0x02d25b to 0x01c8b6 (0x109a4 bytes) 000213 1755038529.014994 mapread 0x0025fb thru 0x00c19f (0x9ba5 bytes) 000214 1755038529.016097 read 0x014305 thru 0x01c8b6 (0x85b2 bytes) 000216 1755038529.017364 write 0x001fb8 thru 0x00e714 (0xc75d bytes) 000217 1755038529.018851 write 0x002409 thru 0x003407 (0xfff bytes) 000219 1755038529.020360 trunc from 0x01c8b7 to 0x03bd02 (0x1f44c bytes) 000220 1755038529.029286 mapwrite 0x037c0d thru 0x03ffff (0x83f3 bytes) 000223 1755038529.080235 mapread 0x03e49b thru 0x03ffff (0x1b65 bytes) 000224 1755038529.081280 read 0x0342bb thru 0x03d283 (0x8fc9 bytes) 000225 1755038529.082089 read 0x009cc0 thru 0x00de23 (0x4164 bytes) 000226 1755038529.082818 read 0x00a860 thru 0x0115fc (0x6d9d bytes) 000229 1755038529.084683 write 0x0029f4 thru 0x005e7b (0x3488 bytes) 000231 1755038529.086351 mapread 0x031794 thru 0x037360 (0x5bcd bytes) 000233 1755038529.090836 MIRROR_EXTEND to 4 mirrors 000235 1755038529.260243 mapread 0x034cf8 thru 0x03ffff (0xb308 bytes) 000236 1755038529.270655 MIRROR_RESYNC 4 mirrors 000238 1755038529.477235 read 0x012b10 thru 0x0225b9 (0xfaaa bytes) 000239 1755038529.482822 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000240 1755038529.578536 trunc from 0x040000 to 0x0305b8 (0xfa47 bytes) 000241 1755038529.611440 mapread 0x0068e9 thru 0x00b27f (0x4997 bytes) 000242 1755038529.618181 read 0x00fb8f thru 0x01ddc4 (0xe236 bytes) 000243 1755038529.622685 read 0x018501 thru 0x02134f (0x8e4f bytes) 000245 1755038529.627386 write 0x023ac7 thru 0x03072f (0xcc69 bytes) 000246 1755038529.628803 trunc from 0x030730 to 0x029d85 (0x69aa bytes) 000249 1755038529.639574 mapread 0x007b4d thru 0x00bb46 (0x3ffa bytes) 000250 1755038529.640458 mapread 0x004df8 thru 0x011116 (0xc31f bytes) 000252 1755038529.644124 MIRROR_EXTEND to 4 mirrors 000253 1755038529.793899 trunc from 0x029d86 to 0x007ee5 (0x21ea0 bytes) skipping zero size read 000256 1755038529.833731 mapread 0x001707 thru 0x0059b1 (0x42ab bytes) 000257 1755038529.841531 read 0x002104 thru 0x007ee5 (0x5de2 bytes) 000258 1755038529.842043 trunc from 0x007ee6 to 0x013d34 (0xbe4f bytes) 000259 1755038529.851053 trunc from 0x013d35 to 0x02e200 (0x1a4cc bytes) 000260 1755038529.858162 trunc from 0x02e201 to 0x005e3a (0x283c6 bytes) 000261 1755038529.866103 mapwrite 0x02a5f3 thru 0x031ed8 (0x78e6 bytes) 000262 1755038529.901686 read 0x0217dc thru 0x026e05 (0x562a bytes) 000264 1755038529.906613 mapwrite 0x031c9c thru 0x03ffff (0xe364 bytes) 000265 1755038529.940415 trunc from 0x040000 to 0x031834 (0xe7cb bytes) 000267 1755038529.948068 mapread 0x02946d thru 0x031834 (0x83c8 bytes) 000269 1755038529.949084 mapwrite 0x0154b3 thru 0x01c75a (0x72a8 bytes) 000271 1755038529.978033 read 0x008f79 thru 0x010455 (0x74dd bytes) 000272 1755038529.979046 mapwrite 0x02f6eb thru 0x036062 (0x6978 bytes) 000273 1755038530.022875 MIRROR_EXTEND to 5 mirrors 000276 1755038530.251967 write 0x019861 thru 0x01dabb (0x425b bytes) 000278 1755038530.287373 write 0x02c827 thru 0x033031 (0x680b bytes) 000279 1755038530.292367 write 0x01e379 thru 0x026306 (0x7f8e bytes) 000281 1755038530.297336 trunc from 0x036063 to 0x010e18 (0x2524a bytes) 000282 1755038530.309290 MIRROR_RESYNC 5 mirrors 000283 1755038530.530476 trunc from 0x010e19 to 0x03d1ce (0x2c3b6 bytes) 000287 1755038530.549341 trunc from 0x03d1cf to 0x018a0a (0x247c4 bytes) 000289 1755038530.556237 trunc from 0x018a0b to 0x02a587 (0x11b7d bytes) 000290 1755038530.560676 write 0x016227 thru 0x01a948 (0x4722 bytes) 000293 1755038530.566091 MIRROR_EXTEND to 6 mirrors 000294 1755038530.700241 mapread 0x007f8e thru 0x00c035 (0x40a8 bytes) 000295 1755038530.706198 write 0x03eade thru 0x03ffff (0x1522 bytes) 000296 1755038530.726750 mapwrite 0x03a04b thru 0x03ffcf (0x5f85 bytes) 000297 1755038530.762476 mapread 0x021a13 thru 0x02e5ea (0xcbd8 bytes) 000299 1755038530.770790 read 0x03e5a9 thru 0x03ffff (0x1a57 bytes) 000302 1755038530.772267 write 0x01e219 thru 0x01f17d (0xf65 bytes) 000303 1755038530.773005 mapread 0x031163 thru 0x03cd1e (0xbbbc bytes) 000304 1755038530.773967 mapread 0x03f5bb thru 0x03ffff (0xa45 bytes) 000306 1755038530.775252 read 0x0071a8 thru 0x008df3 (0x1c4c bytes) 000307 1755038530.779899 mapread 0x012162 thru 0x01e987 (0xc826 bytes) 000309 1755038530.783327 MIRROR_EXTEND to 7 mirrors 000311 1755038530.983209 write 0x030938 thru 0x03ce49 (0xc512 bytes) 000315 1755038531.027230 mapread 0x00ba0a thru 0x00c315 (0x90c bytes) 000317 1755038531.033455 mapread 0x03b6f0 thru 0x03ffff (0x4910 bytes) 000319 1755038531.037669 read 0x03e0a8 thru 0x03ffff (0x1f58 bytes) 000320 1755038531.038280 mapread 0x0040e5 thru 0x00718f (0x30ab bytes) 000321 1755038531.039309 trunc from 0x040000 to 0x00fd8d (0x30272 bytes) 000322 1755038531.058825 read 0x005731 thru 0x00a660 (0x4f30 bytes) 000324 1755038531.060187 write 0x005b0a thru 0x00a58a (0x4a81 bytes) 000326 1755038531.062028 trunc from 0x00fd8e to 0x02cdc3 (0x1d036 bytes) 000327 1755038531.070818 read 0x019014 thru 0x02814b (0xf138 bytes) 000328 1755038531.077146 trunc from 0x02cdc4 to 0x023ab3 (0x9310 bytes) 000330 1755038531.087239 trunc from 0x023ab4 to 0x0278ef (0x3e3c bytes) 000334 1755038531.098315 write 0x01df0c thru 0x0268f8 (0x89ed bytes) 000335 1755038531.102923 read 0x018451 thru 0x01fd54 (0x7904 bytes) 000336 1755038531.103872 mapread 0x00646f thru 0x0074e5 (0x1077 bytes) 000338 1755038531.108593 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000339 1755038531.226030 trunc from 0x0278f0 to 0x0311ac (0x98bd bytes) 000340 1755038531.234024 mapread 0x015e5b thru 0x01e4eb (0x8691 bytes) 000342 1755038531.244357 read 0x012c13 thru 0x01e67b (0xba69 bytes) 000343 1755038531.245119 mapread 0x002b5c thru 0x0126e9 (0xfb8e bytes) 000348 1755038531.252737 mapread 0x000ff1 thru 0x00549c (0x44ac bytes) 000350 1755038531.256794 read 0x02b231 thru 0x030582 (0x5352 bytes) 000351 1755038531.261521 trunc from 0x0311ad to 0x014398 (0x1ce14 bytes) 000352 1755038531.271079 MIRROR_EXTEND to 7 mirrors 000355 1755038531.468128 trunc from 0x014399 to 0x007b42 (0xc856 bytes) 000357 1755038531.525975 mapwrite 0x01f77f thru 0x02acc7 (0xb549 bytes) 000358 1755038531.584532 write 0x010449 thru 0x01c33c (0xbef4 bytes) 000359 1755038531.593792 MIRROR_RESYNC 7 mirrors 000361 1755038532.028461 mapwrite 0x037be8 thru 0x03e083 (0x649c bytes) 000362 1755038532.107920 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000363 1755038532.215103 MIRROR_RESYNC 6 mirrors 000364 1755038532.642375 mapread 0x00365e thru 0x004592 (0xf35 bytes) 000365 1755038532.652496 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000366 1755038532.810336 read 0x00dbd9 thru 0x01080d (0x2c35 bytes) 000370 1755038532.816792 trunc from 0x03e084 to 0x00db8e (0x304f5 bytes) 000371 1755038532.860839 write 0x0165bb thru 0x0182c8 (0x1d0e bytes) 000374 1755038532.863191 mapread 0x009c10 thru 0x00d72a (0x3b1b bytes) 000376 1755038532.875905 mapread 0x016f8f thru 0x0182c8 (0x133a bytes) 000377 1755038532.877244 trunc from 0x0182c9 to 0x037e36 (0x1fb6e bytes) 000378 1755038532.887618 write 0x02e55d thru 0x02fbb2 (0x1656 bytes) 000380 1755038532.896014 MIRROR_RESYNC 5 mirrors 000382 1755038533.254432 mapread 0x01bae4 thru 0x024f5d (0x947a bytes) 000383 1755038533.267170 MIRROR_RESYNC 5 mirrors 000384 1755038533.355107 mapread 0x034914 thru 0x037e36 (0x3523 bytes) 000386 1755038533.369741 MIRROR_RESYNC 5 mirrors 000387 1755038533.481215 mapwrite 0x00a878 thru 0x00cc74 (0x23fd bytes) 000389 1755038533.576870 MIRROR_RESYNC 5 mirrors 000390 1755038533.930147 MIRROR_RESYNC 5 mirrors 000392 1755038534.024014 mapwrite 0x01f1c1 thru 0x021ed8 (0x2d18 bytes) 000393 1755038534.112038 write 0x0231c6 thru 0x0280ec (0x4f27 bytes) 000394 1755038534.113168 trunc from 0x037e37 to 0x010a93 (0x273a3 bytes) 000397 1755038534.123707 read 0x002381 thru 0x006975 (0x45f5 bytes) 000399 1755038534.128353 read 0x010a46 thru 0x010a93 (0x4e bytes) 000400 1755038534.133424 write 0x03fc74 thru 0x03ffff (0x38c bytes) 000401 1755038534.134390 mapread 0x0333d2 thru 0x03ffff (0xcc2e bytes) 000402 1755038534.140488 read 0x038a5d thru 0x03ffff (0x75a3 bytes) 000403 1755038534.141173 trunc from 0x040000 to 0x019b67 (0x26498 bytes) 000404 1755038534.151073 mapread 0x006dcd thru 0x00a2fe (0x3532 bytes) 000405 1755038534.152976 read 0x00c0e5 thru 0x019b67 (0xda83 bytes) 000406 1755038534.154391 read 0x005813 thru 0x00dde2 (0x85d0 bytes) 000407 1755038534.155700 trunc from 0x019b68 to 0x03c0af (0x22548 bytes) 000409 1755038534.165195 mapwrite 0x013a4f thru 0x01b547 (0x7af9 bytes) 000410 1755038534.201959 write 0x02e8f9 thru 0x03506f (0x6777 bytes) 000412 1755038534.203975 read 0x034881 thru 0x03c0af (0x782f bytes) 000415 1755038534.205896 mapwrite 0x00de77 thru 0x0122cc (0x4456 bytes) 000417 1755038534.236666 read 0x002e4c thru 0x00a7b4 (0x7969 bytes) 000418 1755038534.237377 trunc from 0x03c0b0 to 0x01c527 (0x1fb88 bytes) 000419 1755038534.247092 trunc from 0x01c528 to 0x0305f2 (0x140cb bytes) 000421 1755038534.256622 read 0x007f90 thru 0x00ce28 (0x4e99 bytes) 000422 1755038534.260189 MIRROR_EXTEND to 6 mirrors 000426 1755038534.477278 read 0x0084af thru 0x00a92e (0x2480 bytes) 000427 1755038534.483005 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000428 1755038534.614330 mapwrite 0x0353ba thru 0x039e8e (0x4ad5 bytes) 000430 1755038534.707649 write 0x02cbf0 thru 0x0388f9 (0xbd0a bytes) 000432 1755038534.713921 write 0x03d2c0 thru 0x03ffff (0x2d40 bytes) 000433 1755038534.714947 trunc from 0x040000 to 0x02a71c (0x158e3 bytes) 000434 1755038534.726170 mapread 0x000d12 thru 0x0036db (0x29ca bytes) 000437 1755038534.733273 read 0x02175d thru 0x022ead (0x1751 bytes) 000438 1755038534.737156 write 0x021d74 thru 0x030814 (0xeaa1 bytes) 000439 1755038534.738895 read 0x015d4a thru 0x0174fb (0x17b2 bytes) 000444 1755038534.746922 mapwrite 0x01c3b4 thru 0x02b64d (0xf29a bytes) 000446 1755038534.782625 read 0x023ddd thru 0x02f89f (0xbac3 bytes) 000447 1755038534.783391 mapread 0x023a6b thru 0x024e71 (0x1407 bytes) 000451 1755038534.786103 mapwrite 0x02fef1 thru 0x0374b6 (0x75c6 bytes) 000453 1755038534.832171 read 0x01fb51 thru 0x027f74 (0x8424 bytes) 000454 1755038534.832970 mapwrite 0x025db0 thru 0x031f18 (0xc169 bytes) 000455 1755038534.861690 mapread 0x0267ee thru 0x026f95 (0x7a8 bytes) 000457 1755038534.863267 read 0x00677e thru 0x00fb0c (0x938f bytes) 000459 1755038534.864644 read 0x028c6e thru 0x0374b6 (0xe849 bytes) 000461 1755038534.865943 trunc from 0x0374b7 to 0x02e0f9 (0x93bd bytes) 000463 1755038534.875949 trunc from 0x02e0fa to 0x03b71c (0xd623 bytes) 000464 1755038534.884820 trunc from 0x03b71d to 0x013438 (0x282e4 bytes) 000467 1755038534.895244 read 0x002e05 thru 0x00f600 (0xc7fc bytes) 000468 1755038534.896058 trunc from 0x013439 to 0x034472 (0x2103a bytes) 000469 1755038534.905347 read 0x02c320 thru 0x034472 (0x8153 bytes) 000473 1755038534.913832 mapread 0x01ccd7 thru 0x024555 (0x787f bytes) 000477 1755038534.916698 read 0x0007cb thru 0x0024f1 (0x1d27 bytes) 000481 1755038534.919065 write 0x029833 thru 0x02fa3e (0x620c bytes) 000483 1755038534.920933 write 0x022876 thru 0x026af1 (0x427c bytes) 000484 1755038534.922078 write 0x0020f7 thru 0x010ecf (0xedd9 bytes) 000487 1755038534.924753 trunc from 0x034473 to 0x00ab55 (0x2991d bytes) 000488 1755038534.934679 mapread 0x00a1c2 thru 0x00ab55 (0x994 bytes) 000489 1755038534.935709 read 0x004326 thru 0x00ab55 (0x6830 bytes) 000490 1755038534.936403 read 0x002ff3 thru 0x0088e6 (0x58f4 bytes) 000492 1755038534.937866 write 0x017ff3 thru 0x018092 (0xa0 bytes) 000493 1755038534.941871 MIRROR_EXTEND to 6 mirrors 000495 1755038535.199260 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000500 1755038535.385000 mapread 0x00ccc6 thru 0x00e1a4 (0x14df bytes) 000501 1755038535.409824 mapread 0x006fc6 thru 0x0095b9 (0x25f4 bytes) 000504 1755038535.443210 mapread 0x001c0c thru 0x00a894 (0x8c89 bytes) 000505 1755038535.457036 read 0x00d02a thru 0x014fdb (0x7fb2 bytes) 000506 1755038535.466336 mapwrite 0x008237 thru 0x00fed9 (0x7ca3 bytes) 000507 1755038535.553068 trunc from 0x018093 to 0x01b6f8 (0x3666 bytes) 000510 1755038535.567661 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000511 1755038535.689148 write 0x017410 thru 0x01b0b3 (0x3ca4 bytes) 000512 1755038535.695813 read 0x00e2ba thru 0x01b6f8 (0xd43f bytes) 000514 1755038535.704155 MIRROR_EXTEND to 5 mirrors 000515 1755038535.896337 trunc from 0x01b6f9 to 0x0293d2 (0xdcda bytes) 000516 1755038535.953269 mapread 0x017584 thru 0x0226f6 (0xb173 bytes) 000518 1755038535.974312 MIRROR_RESYNC 5 mirrors 000520 1755038536.348893 mapread 0x00d6bd thru 0x014e15 (0x7759 bytes) 000521 1755038536.359744 mapread 0x028797 thru 0x0293d2 (0xc3c bytes) 000522 1755038536.364706 mapread 0x01a43c thru 0x01cdeb (0x29b0 bytes) 000526 1755038536.373254 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000529 1755038536.512176 trunc from 0x0293d3 to 0x00aea6 (0x1e52c bytes) 000533 1755038536.557811 MIRROR_EXTEND to 5 mirrors 000534 1755038536.808294 write 0x028142 thru 0x0304a3 (0x8362 bytes) truncating to largest ever: 0x3e37f 000535 1755038536.848269 trunc from 0x0304a4 to 0x03e37e (0xdedb bytes) 000536 1755038536.861155 trunc from 0x03e37f to 0x02289a (0x1bae4 bytes) 000537 1755038536.870439 mapwrite 0x00a6b5 thru 0x012260 (0x7bac bytes) 000539 1755038536.917075 mapwrite 0x01eb37 thru 0x022000 (0x34ca bytes) 000540 1755038536.950471 write 0x020bcd thru 0x02a003 (0x9437 bytes) 000542 1755038536.952691 mapwrite 0x03c47d thru 0x03ffff (0x3b83 bytes) 000543 1755038536.997786 trunc from 0x040000 to 0x021a21 (0x1e5de bytes) 000546 1755038537.008779 mapwrite 0x02f3ce thru 0x03b84f (0xc482 bytes) 000547 1755038537.055988 read 0x024bdd thru 0x0335ab (0xe9cf bytes) 000549 1755038537.060734 read 0x025a91 thru 0x028198 (0x2708 bytes) 000550 1755038537.061431 mapread 0x00c89f thru 0x017888 (0xafea bytes) 000552 1755038537.068172 trunc from 0x03b850 to 0x00e8fb (0x2cf54 bytes) 000553 1755038537.078109 read 0x001dde thru 0x00c280 (0xa4a3 bytes) 000555 1755038537.079466 read 0x00a577 thru 0x00c8ae (0x2338 bytes) 000556 1755038537.080216 write 0x025751 thru 0x02fb59 (0xa409 bytes) 000557 1755038537.081872 write 0x014e8d thru 0x021475 (0xc5e9 bytes) 000558 1755038537.088198 write 0x003894 thru 0x00b8bd (0x802a bytes) 000559 1755038537.089516 read 0x001f6f thru 0x00e595 (0xc627 bytes) 000560 1755038537.090226 write 0x0303bb thru 0x03d65a (0xd2a0 bytes) 000561 1755038537.091839 write 0x032b0f thru 0x03ffff (0xd4f1 bytes) 000566 1755038537.097330 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000568 1755038537.180394 MIRROR_EXTEND to 5 mirrors 000569 1755038537.417631 MIRROR_RESYNC 5 mirrors 000571 1755038537.741861 mapread 0x022ac8 thru 0x02da18 (0xaf51 bytes) 000573 1755038537.758236 trunc from 0x040000 to 0x034c97 (0xb368 bytes) 000574 1755038537.808304 mapwrite 0x03e04d thru 0x03ffff (0x1fb3 bytes) 000575 1755038537.853442 mapwrite 0x017602 thru 0x022f72 (0xb971 bytes) 000576 1755038537.893537 trunc from 0x040000 to 0x00c8da (0x33725 bytes) 000577 1755038537.903923 read 0x00b46e thru 0x00c8da (0x146d bytes) 000579 1755038537.908424 mapwrite 0x0342ca thru 0x03499c (0x6d3 bytes) 000581 1755038537.950620 mapwrite 0x007698 thru 0x00f1fc (0x7b65 bytes) 000585 1755038537.987023 read 0x021adc thru 0x027ed6 (0x63fb bytes) 000586 1755038537.993454 write 0x015707 thru 0x02523e (0xfb38 bytes) 000587 1755038537.997971 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000588 1755038538.110121 MIRROR_RESYNC 4 mirrors 000589 1755038538.444698 MIRROR_RESYNC 4 mirrors 000590 1755038538.521273 MIRROR_RESYNC 4 mirrors 000593 1755038538.609175 mapwrite 0x036f22 thru 0x03d72a (0x6809 bytes) 000594 1755038538.689911 mapread 0x00fc66 thru 0x01df48 (0xe2e3 bytes) 000595 1755038538.700169 write 0x02349a thru 0x02e9ac (0xb513 bytes) 000596 1755038538.708792 trunc from 0x03d72b to 0x0226bc (0x1b06e bytes) 000598 1755038538.721829 mapread 0x015773 thru 0x01d01b (0x78a9 bytes) 000599 1755038538.722995 mapread 0x0031d1 thru 0x006749 (0x3579 bytes) 000600 1755038538.728856 write 0x000efa thru 0x008630 (0x7737 bytes) 000601 1755038538.732991 read 0x00bddb thru 0x00cdd3 (0xff9 bytes) 000603 1755038538.734282 mapwrite 0x02e051 thru 0x03621a (0x81ca bytes) 000607 1755038538.783097 trunc from 0x03621b to 0x00ee45 (0x273d5 bytes) 000608 1755038538.793079 mapread 0x003e97 thru 0x00dde2 (0x9f4c bytes) 000609 1755038538.794196 mapread 0x00b87b thru 0x00ee45 (0x35cb bytes) 000612 1755038538.799242 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000613 1755038538.908063 write 0x01154a thru 0x01ae10 (0x98c7 bytes) 000614 1755038538.909798 read 0x006f79 thru 0x0154ed (0xe575 bytes) 000615 1755038538.915395 mapread 0x00786d thru 0x0142ff (0xca93 bytes) 000616 1755038538.916798 read 0x00e61a thru 0x019e7e (0xb865 bytes) 000617 1755038538.917648 read 0x000439 thru 0x00bfa6 (0xbb6e bytes) 000618 1755038538.922987 mapwrite 0x026d5c thru 0x0350ea (0xe38f bytes) 000620 1755038538.976589 MIRROR_RESYNC 3 mirrors 000622 1755038539.256902 read 0x027f92 thru 0x02beda (0x3f49 bytes) 000625 1755038539.262170 read 0x0130e3 thru 0x0193d5 (0x62f3 bytes) 000628 1755038539.271643 MIRROR_RESYNC 3 mirrors 000629 1755038539.356426 MIRROR_EXTEND to 4 mirrors 000634 1755038539.609967 mapwrite 0x00d0f0 thru 0x017073 (0x9f84 bytes) 000635 1755038539.711641 MIRROR_RESYNC 4 mirrors 000637 1755038539.977096 mapread 0x02d2db thru 0x0350ea (0x7e10 bytes) 000638 1755038539.982888 read 0x02cb2d thru 0x033aac (0x6f80 bytes) 000639 1755038539.984387 MIRROR_EXTEND to 5 mirrors 000640 1755038540.135735 mapwrite 0x003841 thru 0x00914a (0x590a bytes) 000641 1755038540.199909 write 0x0228ea thru 0x02c378 (0x9a8f bytes) 000642 1755038540.208892 mapwrite 0x0114ec thru 0x01820e (0x6d23 bytes) 000643 1755038540.243143 MIRROR_EXTEND to 6 mirrors 000644 1755038540.453847 trunc from 0x0350eb to 0x03e138 (0x904e bytes) 000645 1755038540.507668 write 0x03cf81 thru 0x03ea90 (0x1b10 bytes) 000649 1755038540.516171 trunc from 0x03ea91 to 0x01baae (0x22fe2 bytes) 000650 1755038540.528387 trunc from 0x01baaf to 0x03b179 (0x1f6cb bytes) 000652 1755038540.538190 read 0x02a5e7 thru 0x02ace5 (0x6ff bytes) 000654 1755038540.542732 mapread 0x02f3a6 thru 0x03adf9 (0xba54 bytes) 000655 1755038540.552452 read 0x01cdb6 thru 0x029f1e (0xd169 bytes) 000658 1755038540.563121 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000659 1755038540.673922 write 0x037f20 thru 0x03ffff (0x80e0 bytes) 000663 1755038540.679945 trunc from 0x040000 to 0x00592d (0x3a6d2 bytes) 000664 1755038540.689295 write 0x02e614 thru 0x0344ce (0x5ebb bytes) 000665 1755038540.690832 read 0x01957e thru 0x02696f (0xd3f2 bytes) 000666 1755038540.696895 mapread 0x002687 thru 0x00ed48 (0xc6c2 bytes) 000668 1755038540.707293 read 0x00de6c thru 0x019a1d (0xbbb2 bytes) 000669 1755038540.713950 mapread 0x031890 thru 0x0344ce (0x2c3f bytes) 000671 1755038540.716794 write 0x01ef63 thru 0x02cd86 (0xde24 bytes) 000672 1755038540.719519 read 0x0203b2 thru 0x02175e (0x13ad bytes) 000676 1755038540.723492 read 0x013344 thru 0x014651 (0x130e bytes) 000679 1755038540.730076 MIRROR_EXTEND to 6 mirrors 000680 1755038540.955502 MIRROR_EXTEND to 7 mirrors 000681 1755038541.209471 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000682 1755038541.337212 write 0x025a67 thru 0x031ba6 (0xc140 bytes) 000684 1755038541.383709 mapread 0x02dbba thru 0x030c82 (0x30c9 bytes) 000685 1755038541.384869 write 0x02a0a2 thru 0x036e73 (0xcdd2 bytes) 000686 1755038541.386579 read 0x00a3e4 thru 0x0131da (0x8df7 bytes) 000689 1755038541.393601 write 0x0288e7 thru 0x035796 (0xceb0 bytes) 000690 1755038541.398098 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000691 1755038541.513720 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000692 1755038541.614365 read 0x000fef thru 0x00477c (0x378e bytes) 000693 1755038541.621300 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000694 1755038541.722766 mapread 0x018301 thru 0x023a24 (0xb724 bytes) 000695 1755038541.741117 MIRROR_RESYNC 3 mirrors 000696 1755038542.012090 MIRROR_RESYNC 3 mirrors 000697 1755038542.095496 read 0x00ca23 thru 0x00f622 (0x2c00 bytes) 000699 1755038542.100296 mapread 0x007527 thru 0x01333f (0xbe19 bytes) 000700 1755038542.116506 mapread 0x00d149 thru 0x00e59e (0x1456 bytes) 000701 1755038542.117254 mapwrite 0x030812 thru 0x034dc4 (0x45b3 bytes) 000703 1755038542.203650 trunc from 0x036e74 to 0x0121a0 (0x24cd3 bytes) 000705 1755038542.226245 read 0x00957b thru 0x0121a0 (0x8c26 bytes) 000706 1755038542.231950 mapread 0x0073b2 thru 0x00d327 (0x5f76 bytes) 000707 1755038542.233084 write 0x03de6c thru 0x03dee7 (0x7c bytes) 000708 1755038542.234244 mapwrite 0x031d0b thru 0x0339c4 (0x1cba bytes) 000710 1755038542.269843 mapread 0x0356e2 thru 0x03dee7 (0x8806 bytes) 000713 1755038542.274050 read 0x00072d thru 0x009f8d (0x9861 bytes) 000714 1755038542.275317 mapwrite 0x00f970 thru 0x0185ca (0x8c5b bytes) 000715 1755038542.305201 mapread 0x035714 thru 0x03580f (0xfc bytes) 000719 1755038542.308044 mapread 0x0152b8 thru 0x015311 (0x5a bytes) 000721 1755038542.309620 write 0x018775 thru 0x01bede (0x376a bytes) 000722 1755038542.313603 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000723 1755038542.422027 write 0x027d05 thru 0x035e7f (0xe17b bytes) 000724 1755038542.429529 mapread 0x015c79 thru 0x0169b7 (0xd3f bytes) 000728 1755038542.436953 mapread 0x0014cc thru 0x00be26 (0xa95b bytes) 000730 1755038542.443605 mapread 0x0151c8 thru 0x0174ca (0x2303 bytes) 000732 1755038542.445287 write 0x03a165 thru 0x03ffff (0x5e9b bytes) 000733 1755038542.450949 mapread 0x01b299 thru 0x0275d4 (0xc33c bytes) 000735 1755038542.457201 mapwrite 0x0130a3 thru 0x0172a0 (0x41fe bytes) 000737 1755038542.493166 MIRROR_EXTEND to 3 mirrors 000738 1755038542.716179 mapread 0x011306 thru 0x01fcf0 (0xe9eb bytes) 000739 1755038542.735682 MIRROR_EXTEND to 4 mirrors 000741 1755038542.979894 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000742 1755038543.112997 write 0x00c9ae thru 0x016aa2 (0xa0f5 bytes) 000743 1755038543.159276 mapread 0x02e15f thru 0x02f490 (0x1332 bytes) 000744 1755038543.165875 trunc from 0x040000 to 0x02b8ef (0x14710 bytes) 000745 1755038543.187530 mapread 0x00bc5c thru 0x00d2da (0x167f bytes) 000746 1755038543.196410 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000747 1755038543.304860 mapwrite 0x03edd4 thru 0x03ffff (0x122c bytes) 000748 1755038543.351061 mapwrite 0x0071ef thru 0x0085df (0x13f1 bytes) 000750 1755038543.381749 write 0x026e83 thru 0x02d46c (0x65ea bytes) 000751 1755038543.388020 mapwrite 0x01164f thru 0x014520 (0x2ed2 bytes) 000752 1755038543.424938 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000755 1755038543.499475 trunc from 0x040000 to 0x00806d (0x37f92 bytes) 000757 1755038543.509312 read 0x000531 thru 0x00806d (0x7b3d bytes) 000759 1755038543.515286 mapwrite 0x02e4d8 thru 0x03212a (0x3c53 bytes) 000761 1755038543.568509 read 0x01dba8 thru 0x01fdb2 (0x220b bytes) 000764 1755038543.576964 read 0x002b28 thru 0x002ccf (0x1a8 bytes) 000766 1755038543.581254 MIRROR_EXTEND to 2 mirrors 000767 1755038543.785215 read 0x02a65b thru 0x03212a (0x7ad0 bytes) 000768 1755038543.789700 write 0x02f73d thru 0x034ae7 (0x53ab bytes) 000769 1755038543.831953 trunc from 0x034ae8 to 0x02fa1d (0x50ca bytes) 000770 1755038543.851279 read 0x0005a6 thru 0x0027af (0x220a bytes) 000771 1755038543.858020 mapwrite 0x002654 thru 0x003b5c (0x1509 bytes) 000772 1755038543.882749 write 0x038478 thru 0x03a749 (0x22d2 bytes) 000774 1755038543.886490 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000775 1755038543.954859 mapwrite 0x004f9d thru 0x01280d (0xd871 bytes) 000777 1755038544.001804 MIRROR_EXTEND to 2 mirrors 000779 1755038544.256768 trunc from 0x03a74a to 0x01d792 (0x1cfb7 bytes) 000780 1755038544.302221 mapwrite 0x03d4b2 thru 0x03ffff (0x2b4e bytes) 000781 1755038544.354520 read 0x010811 thru 0x01670f (0x5eff bytes) 000782 1755038544.360041 mapwrite 0x0180b9 thru 0x01e871 (0x67b9 bytes) 000783 1755038544.392338 read 0x029699 thru 0x0348fe (0xb266 bytes) 000786 1755038544.400200 mapwrite 0x03595e thru 0x03ffff (0xa6a2 bytes) 000787 1755038544.427985 write 0x02d442 thru 0x035840 (0x83ff bytes) 000788 1755038544.429317 write 0x018448 thru 0x0280f8 (0xfcb1 bytes) 000790 1755038544.431504 read 0x0013ae thru 0x001ce7 (0x93a bytes) 000794 1755038544.434038 write 0x021039 thru 0x022fef (0x1fb7 bytes) 000795 1755038544.437771 MIRROR_EXTEND to 3 mirrors 000799 1755038544.622560 trunc from 0x040000 to 0x017d46 (0x282b9 bytes) 000800 1755038544.655412 trunc from 0x017d47 to 0x005665 (0x126e1 bytes) 000802 1755038544.667600 MIRROR_EXTEND to 4 mirrors 000803 1755038544.871311 read 0x0053a5 thru 0x005665 (0x2c1 bytes) 000804 1755038544.877438 mapread 0x00335d thru 0x004e09 (0x1aad bytes) 000805 1755038544.888495 write 0x00e932 thru 0x014bbe (0x628d bytes) 000806 1755038544.927069 mapwrite 0x015719 thru 0x02371e (0xe006 bytes) 000807 1755038544.995093 mapwrite 0x02ad92 thru 0x02e165 (0x33d4 bytes) 000808 1755038545.040733 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000809 1755038545.147015 read 0x022ca6 thru 0x0233f6 (0x751 bytes) 000810 1755038545.150905 mapwrite 0x024c29 thru 0x03337c (0xe754 bytes) 000811 1755038545.208239 read 0x00efbe thru 0x0173ab (0x83ee bytes) 000813 1755038545.215216 trunc from 0x03337d to 0x01bda0 (0x175dc bytes) 000816 1755038545.226726 trunc from 0x01bda1 to 0x020dcf (0x502f bytes) 000817 1755038545.235509 mapread 0x003934 thru 0x00e631 (0xacfe bytes) 000818 1755038545.236775 write 0x02becf thru 0x03ad92 (0xeec4 bytes) 000819 1755038545.238704 mapwrite 0x0238b7 thru 0x027a89 (0x41d3 bytes) 000820 1755038545.287643 mapwrite 0x002286 thru 0x0054b0 (0x322b bytes) 000821 1755038545.311184 mapwrite 0x028f62 thru 0x02dd29 (0x4dc8 bytes) 000823 1755038545.350118 MIRROR_RESYNC 3 mirrors 000826 1755038545.618443 mapwrite 0x000c46 thru 0x0069b7 (0x5d72 bytes) 000828 1755038545.712953 write 0x031dd0 thru 0x039abb (0x7cec bytes) 000829 1755038545.722512 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000831 1755038545.829548 trunc from 0x03ad93 to 0x027f75 (0x12e1d bytes) 000832 1755038545.847385 mapread 0x00c635 thru 0x00d94d (0x1319 bytes) 000833 1755038545.854539 mapwrite 0x001b77 thru 0x00f043 (0xd4cd bytes) 000834 1755038545.893307 mapwrite 0x005453 thru 0x00d41e (0x7fcc bytes) 000835 1755038545.921540 trunc from 0x027f76 to 0x018459 (0xfb1c bytes) 000836 1755038545.931379 read 0x0107ec thru 0x018459 (0x7c6e bytes) 000839 1755038545.938125 read 0x010c3b thru 0x018459 (0x781f bytes) 000841 1755038545.939476 mapwrite 0x03ebbb thru 0x03ffff (0x1445 bytes) 000843 1755038545.981359 write 0x023eb2 thru 0x027cb9 (0x3e08 bytes) 000844 1755038545.982670 read 0x019cad thru 0x028c8e (0xefe2 bytes) 000848 1755038545.985270 write 0x020500 thru 0x02ed86 (0xe887 bytes) 000850 1755038545.987426 trunc from 0x040000 to 0x03332b (0xccd4 bytes) 000851 1755038545.996962 read 0x00dfa3 thru 0x0192f5 (0xb353 bytes) 000852 1755038545.997761 trunc from 0x03332c to 0x0169d7 (0x1c954 bytes) 000853 1755038546.010119 MIRROR_EXTEND to 3 mirrors 000854 1755038546.249832 mapwrite 0x03dd7f thru 0x03ffff (0x2281 bytes) 000855 1755038546.331697 read 0x03d50a thru 0x03ffb4 (0x2aab bytes) 000856 1755038546.332518 mapwrite 0x015fe5 thru 0x018e12 (0x2e2e bytes) 000857 1755038546.360777 mapwrite 0x0078cf thru 0x00abe3 (0x3315 bytes) 000859 1755038546.390243 trunc from 0x040000 to 0x00e677 (0x31988 bytes) 000860 1755038546.401756 read 0x00bd8f thru 0x00e677 (0x28e9 bytes) 000862 1755038546.402927 write 0x0009ab thru 0x00601e (0x5674 bytes) 000865 1755038546.407937 read 0x00a22d thru 0x00e677 (0x444b bytes) 000866 1755038546.408676 write 0x023e9b thru 0x02e2d0 (0xa436 bytes) 000867 1755038546.410077 mapwrite 0x016d36 thru 0x01cbbc (0x5e87 bytes) 000868 1755038546.449282 mapwrite 0x038a96 thru 0x03ffff (0x756a bytes) 000869 1755038546.495510 trunc from 0x040000 to 0x029ecc (0x16133 bytes) 000871 1755038546.505408 write 0x02fa49 thru 0x0308ad (0xe65 bytes) 000872 1755038546.506414 read 0x029bc0 thru 0x0308ad (0x6cee bytes) 000873 1755038546.511090 write 0x0151ec thru 0x01bcec (0x6b01 bytes) 000874 1755038546.512070 trunc from 0x0308ae to 0x012668 (0x1e245 bytes) 000875 1755038546.520377 write 0x00816d thru 0x00e35d (0x61f1 bytes) 000876 1755038546.521749 mapwrite 0x002646 thru 0x00a639 (0x7ff4 bytes) 000877 1755038546.548453 write 0x0148ec thru 0x01a264 (0x5979 bytes) 000878 1755038546.549910 read 0x00cefe thru 0x01a124 (0xd227 bytes) 000879 1755038546.556250 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000880 1755038546.665524 write 0x01e306 thru 0x0249eb (0x66e6 bytes) 000881 1755038546.667015 read 0x012e83 thru 0x019f61 (0x70df bytes) 000883 1755038546.672947 read 0x0160a5 thru 0x022bb1 (0xcb0d bytes) 000884 1755038546.679397 write 0x01f8b4 thru 0x027d4b (0x8498 bytes) 000885 1755038546.680964 mapread 0x009c12 thru 0x00e7ef (0x4bde bytes) 000886 1755038546.681980 write 0x027c5f thru 0x03107f (0x9421 bytes) 000889 1755038546.684886 write 0x013f82 thru 0x01d2c5 (0x9344 bytes) 000890 1755038546.686362 trunc from 0x031080 to 0x006213 (0x2ae6c bytes) 000893 1755038546.697302 mapwrite 0x03d2cc thru 0x03ffff (0x2d34 bytes) 000895 1755038546.741428 trunc from 0x040000 to 0x023105 (0x1cefa bytes) 000897 1755038546.751675 mapwrite 0x01bd00 thru 0x01c638 (0x939 bytes) 000898 1755038546.789525 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000902 1755038546.859267 read 0x005458 thru 0x00b6dc (0x6285 bytes) 000903 1755038546.864763 mapread 0x015316 thru 0x0206a2 (0xb38d bytes) 000904 1755038546.879181 trunc from 0x023106 to 0x02518c (0x2087 bytes) 000906 1755038546.888251 mapwrite 0x015669 thru 0x021a09 (0xc3a1 bytes) 000908 1755038546.923923 trunc from 0x02518d to 0x00ef03 (0x16289 bytes) 000911 1755038546.935794 trunc from 0x00ef04 to 0x004932 (0xa5d1 bytes) 000912 1755038546.945080 write 0x00703d thru 0x00825a (0x121e bytes) 000913 1755038546.946181 mapread 0x004a38 thru 0x00825a (0x3823 bytes) 000914 1755038546.951514 mapread 0x00300c thru 0x00825a (0x524f bytes) 000916 1755038546.953262 mapread 0x000d7d thru 0x00825a (0x74de bytes) 000917 1755038546.956859 read 0x00065d thru 0x00825a (0x7bfe bytes) 000919 1755038546.957892 trunc from 0x00825b to 0x02793e (0x1f6e4 bytes) 000922 1755038546.968279 read 0x00c4ba thru 0x012817 (0x635e bytes) 000923 1755038546.975015 mapwrite 0x01dd66 thru 0x02503e (0x72d9 bytes) 000926 1755038547.010487 mapwrite 0x0109d7 thru 0x01df0d (0xd537 bytes) 000934 1755038547.046420 trunc from 0x02793f to 0x019aa2 (0xde9c bytes) 000936 1755038547.057900 mapwrite 0x028b67 thru 0x032640 (0x9ada bytes) 000938 1755038547.105924 trunc from 0x032641 to 0x012cfd (0x1f943 bytes) 000939 1755038547.115796 mapwrite 0x0111e1 thru 0x01c62b (0xb44b bytes) 000940 1755038547.187333 mapread 0x008a5b thru 0x014020 (0xb5c6 bytes) 000941 1755038547.188644 read 0x01abe2 thru 0x01c62b (0x1a4a bytes) 000943 1755038547.189958 mapread 0x00f820 thru 0x010fe4 (0x17c5 bytes) 000945 1755038547.191464 trunc from 0x01c62c to 0x00b701 (0x10f2a bytes) 000947 1755038547.202079 write 0x020e6b thru 0x02c8df (0xba75 bytes) 000948 1755038547.203857 write 0x020aec thru 0x0263d5 (0x58ea bytes) 000952 1755038547.206805 mapwrite 0x02b979 thru 0x037baf (0xc237 bytes) 000955 1755038547.258678 mapread 0x00489a thru 0x00f980 (0xb0e7 bytes) 000957 1755038547.260386 write 0x0388a7 thru 0x039333 (0xa8d bytes) 000958 1755038547.261484 trunc from 0x039334 to 0x023312 (0x16021 bytes) 000959 1755038547.271738 trunc from 0x023313 to 0x00e22f (0x150e3 bytes) 000961 1755038547.284862 MIRROR_EXTEND to 2 mirrors 000963 1755038547.520147 read 0x00c0dd thru 0x00e22f (0x2153 bytes) 000964 1755038547.526475 MIRROR_EXTEND to 3 mirrors 000965 1755038547.768905 read 0x00d6c0 thru 0x00e22f (0xb70 bytes) 000966 1755038547.775043 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000967 1755038547.894947 write 0x013127 thru 0x01ae5b (0x7d35 bytes) 000969 1755038547.934672 mapwrite 0x02562b thru 0x03167d (0xc053 bytes) 000970 1755038548.001075 MIRROR_RESYNC 2 mirrors 000971 1755038548.230238 write 0x00036a thru 0x00e9e2 (0xe679 bytes) 000974 1755038548.270850 trunc from 0x03167e to 0x0176c9 (0x19fb4 bytes) 000976 1755038548.283228 trunc from 0x0176ca to 0x01ff94 (0x88cb bytes) 000977 1755038548.295315 MIRROR_EXTEND to 3 mirrors 000978 1755038548.538866 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3dc:0x0]: Invalid argument (22) 000979 1755038548.654739 read 0x0125db thru 0x01c3f7 (0x9e1d bytes) 000981 1755038548.662199 read 0x00d49a thru 0x010615 (0x317c bytes) 000982 1755038548.668221 mapwrite 0x0280f5 thru 0x03469c (0xc5a8 bytes) 000983 1755038548.754428 read 0x00406d thru 0x008a5c (0x49f0 bytes) 000984 1755038548.760869 write 0x00cd8e thru 0x00df65 (0x11d8 bytes) 000988 1755038548.763707 mapread 0x02c54e thru 0x034197 (0x7c4a bytes) 000989 1755038548.764736 write 0x0203e1 thru 0x0253b9 (0x4fd9 bytes) 000992 1755038548.766887 mapwrite 0x02b984 thru 0x030549 (0x4bc6 bytes) 000993 1755038548.796604 mapwrite 0x035d99 thru 0x03ffff (0xa267 bytes) 000995 1755038548.844619 read 0x01904d thru 0x01ffbe (0x6f72 bytes) 000997 1755038548.845989 mapwrite 0x0019ac thru 0x0048ec (0x2f41 bytes) 000999 1755038548.869919 trunc from 0x040000 to 0x02442e (0x1bbd1 bytes) All operations completed A-OK! PASS 70a (27s) == sanity-flr test 71: check mirror extend parent fid ==== 18:42:31 (1755038551) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 18:42:33 (1755038553) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre2 Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6362368 .. Extending file size to 2457824 .. failed Extending file size to 7798688 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3351776 .. Extending file size to 5955808 .. Extending file size to 7845280 .. Extending file size to 8929536 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8178656 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8671008 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8710304 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8937600 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9023328 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56539 56540 56542 56544 56546 done 192.168.204.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre2 (opts:) 192.168.204.125@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (67s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:43:40 (1755038620) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre2 Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3389184 .. Extending file size to 740992 .. Extending file size to 6356384 .. Extending file size to 5208896 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7106272 .. done Extending file size to 6541088 .. Extending file size to 7094848 .. Extending file size to 8289824 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7235872 .. failed Extending file size to 8607168 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8578752 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8612448 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8788896 .. Extending file size to 8835648 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8742016 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9041536 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8977408 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9204736 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9180768 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59692 59693 59695 59697 59699 done 192.168.204.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre2 (opts:) 192.168.204.125@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:44:47 (1755038687) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4480 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 19456 3749888 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4480 2203776 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 19456 3749888 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.044422 s, 47.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100028 s, 21.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:44:55 (1755038695) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0423257 s, 49.5 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 18:44:58 (1755038698) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x429:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fd:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fd:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5fe:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x5fa:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:45:01 (1755038701) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0503353 s, 41.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ff:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fc:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:45:04 (1755038704) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0570456 s, 36.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x600:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x601:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x600:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x601:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5fe:0x0] } PASS 204b (4s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:45:08 (1755038708) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0434077 s, 48.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x602:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x602:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x600:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:45:33 (1755038733) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg425-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03945 s, 26.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x603:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x602:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x601:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x603:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x602:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x601:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x604:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:46:19 (1755038779) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg425-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x603:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x604:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0331103 s, 63.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x603:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x604:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x605:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x603:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x604:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x605:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:47:05 (1755038825) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg425-server mds-ost sync done. Creating new pool test_204f oleg425-server: Pool lustre.test_204f created Adding targets to pool oleg425-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg425-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0276522 s, 75.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x605:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x606:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x605:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x606:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg425-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg425-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg425-server: Pool lustre.test_204f destroyed Waiting 90s for 'foo' PASS 204f (58s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:48:03 (1755038883) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:48:05 (1755038885) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:48:08 (1755038888) oleg425-server: Pool lustre.test_206 created Adding targets to pool oleg425-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg425-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg425-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg425-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg425-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:48:23 (1755038903) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345727 s, 30.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017725 s, 59.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xe3166947 Mirror 3: 0xe3166947 PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:48:27 (1755038907) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:48:28 (1755038908) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:48:29 (1755038909) Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0182126 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2460000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xfd0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xfd0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xfd0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xf50000 Read this (4 bytes): BBBB 192.168.204.125@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg425-client.virtnet /mnt/lustre2 (opts:) PASS 209a (17s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:48:46 (1755038926) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00487112 s, 1.7 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:48:49 (1755038929) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259011 s, 40.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x44e lmm_fid: [0x200000404:0x44e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x44e lmm_fid: [0x200000404:0x44e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x610:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835022 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-08-12 18:48:50.000000000 -0400 Modify: 2025-08-12 18:48:50.000000000 -0400 Change: 2025-08-12 18:48:50.000000000 -0400 Birth: 2025-08-12 18:48:50.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:48:53 (1755038933) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:48:54 (1755038934) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.180769 s, 227 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0249172 s, 164 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 12 18:48 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x450:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 12 18:48 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: mirror copy preserves exact content for various file sizes ========================================================== 18:48:58 (1755038938) Testing mirror copy with 1 bytes (page_size=4096) mirror copied successfully: 2 Testing mirror copy with 4096 bytes (page_size=4096) mirror copied successfully: 2 Testing mirror copy with 4095 bytes (page_size=4096) mirror copied successfully: 2 Testing mirror copy with 4097 bytes (page_size=4096) mirror copied successfully: 2 Testing mirror copy with 4113 bytes (page_size=4096) mirror copied successfully: 2 Testing mirror copy with 8192 bytes (page_size=4096) mirror copied successfully: 2 Testing mirror copy with 8189 bytes (page_size=4096) mirror copied successfully: 2 Testing mirror copy with 12295 bytes (page_size=4096) mirror copied successfully: 2 PASS 212a (158s) == sanity-flr test 212b: mirror copy preserves data integrity at page boundaries ========================================================== 18:51:36 (1755039096) Testing mirror copy with data at page boundaries mirror copied successfully: 2 Page boundary test passed - no corruption detected PASS 212b (3s) == sanity-flr test complete, duration 982 sec ============ 18:51:39 (1755039099) === sanity-flr: start cleanup 18:51:40 (1755039100) === === sanity-flr: finish cleanup 18:51:42 (1755039102) ===