-----============= acceptance-small: sanity-flr ============----- Thu Nov 6 23:20:49 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg413-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:21:11 (1762489271) === oleg413-client.virtnet: executing check_config_client /mnt/lustre oleg413-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg413-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b7146718000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b7146718000.idle_timeout=debug disable quota as required oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:21:43 (1762489303) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7566] == sanity-flr test 0a: lfs mirror create with -N option == 23:21:45 (1762489305) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:22:04 (1762489324) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:22:08 (1762489328) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:22:13 (1762489333) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:22:38 (1762489358) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:22:42 (1762489362) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 23:22:45 (1762489365) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b7146718000.stats=clear osc.lustre-OST0001-osc-ffff9b7146718000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:22:53 (1762489373) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:23:03 (1762489383) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:23:11 (1762489391) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:23:14 (1762489394) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 23:23:23 (1762489403) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:23:31 (1762489411) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:23:34 (1762489414) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:23:40 (1762489420) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:23:50 (1762489430) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:23:56 (1762489436) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.95951 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157808 s, 6.6 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:24:08 (1762489448) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.407431 s, 25.7 MB/s PASS 22 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:24:16 (1762489456) Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b7146718000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b7146718000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b7146718000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b7146718000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (66s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:25:22 (1762489522) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.648347 s, 16.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b7146718000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b7146718000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (37s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:25:59 (1762489559) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b7146718000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b7146718000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b7146718000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b7146718000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:27:14 (1762489634) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.99223 s, 13.1 MB/s -rw-r--r-- 1 root root 52428800 Nov 6 23:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.003s user 0m0.000s sys 0m0.172s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b7146718000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b7146718000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.235s user 0m0.001s sys 0m0.218s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b7146718000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b7146718000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.136s user 0m0.000s sys 0m0.177s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b7146718000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b7146718000.ost_server_uuid in FULL state after 0 sec PASS 33b (75s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:28:30 (1762489710) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:28:33 (1762489713) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:28:36 (1762489716) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 23:28:39 (1762489719) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 23:28:46 (1762489726) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.474625 s, 26.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:28:58 (1762489738) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260758 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268217 s, 39.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:29:05 (1762489745) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:29:13 (1762489753) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00112152 s, 913 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00267312 s, 383 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00144105 s, 711 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109465 s, 935 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 23:29:23 (1762489763) create test files with size 15733837 .. 51f0bc39d4af9582e4cf84a7c052d4ba - 54a8dbb9642fff4692ba2bc8c53dbd6d - f7981df5484bc935ed2407a643bb2724 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 23:29:48 (1762489788) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:30:01 (1762489801) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:30:07 (1762489807) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13103 s, 8.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x49:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4a:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x45:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492512 s, 21.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x46:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 23:30:16 (1762489816) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116118 s, 18.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.386595 s, 10.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155155 s, 13.5 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (11s) == sanity-flr test 42: lfs mirror verify ================= 23:30:28 (1762489828) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 23:30:31 (1762489831) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:30:34 (1762489834) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240299 s, 43.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 23:30:40 (1762489840) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:30:43 (1762489843) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x4e:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x52:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:30:52 (1762489852) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.190911 s, 54.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:31:01 (1762489861) llite.lustre-ffff9b7147cf4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.319359 s, 32.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b715043d800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:31:12 (1762489872) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:31:48 (1762489908) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:32:21 (1762489941) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 23:32:24 (1762489944) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.330723 s, 9.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.146674 s, 21.4 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 8a16b673a2d7e0d9faf5cf97ec50b254 - mirror 1 checksum: f8f7c3f142f8f28648a9c5baeb622a29 - mirror 2 checksum: 8a16b673a2d7e0d9faf5cf97ec50b254 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Nov 6 23:32:28 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Nov 6 23:32:28 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8a16b673a2d7e0d9faf5cf97ec50b254 - mirror 1 checksum: f8f7c3f142f8f28648a9c5baeb622a29 - mirror 2 checksum: f8f7c3f142f8f28648a9c5baeb622a29 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Nov 6 23:32:28 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x64:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Nov 6 23:32:28 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x65:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:32:35 (1762489955) Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 23:32:41 (1762489961) shut down facet: mds1 facet_host: oleg413-server facet_failover_host: oleg413-server Failover mds1 to oleg413-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 23:33:03 (1762489983) targets are mounted 23:33:03 (1762489983) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290834 s, 36.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } PASS 50A (42s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:33:17 (1762489997) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:33:20 (1762490000) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:33:23 (1762490003) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00591745 s, 2.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:33:31 (1762490011) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:33:34 (1762490014) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:33:36 (1762490016) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:33:57 (1762490037) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:34:26 (1762490066) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:34:50 (1762490090) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:35:14 (1762490114) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 23:35:22 (1762490122) mirror create pid 52284 mirror split pid 52285 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 52284 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 52285 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 23:36:28 (1762490188) Seed set to 190 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x7b68 000001 1762490190.787221 trunc from 00000000 to 0x007b67 (0x7b68 bytes) 000002 1762490190.802336 write 0x0240bf thru 0x02c01b (0x7f5d bytes) truncating to largest ever: 0x18d75 000003 1762490190.803689 trunc from 0x02c01c to 0x018d74 (0x132a7 bytes) 000004 1762490190.830489 read 0x008206 thru 0x00e31b (0x6116 bytes) truncating to largest ever: 0x3d8d0 000005 1762490190.840177 trunc from 0x018d75 to 0x03d8cf (0x24b5b bytes) 000006 1762490190.861359 mapwrite 0x03b951 thru 0x03ffff (0x46af bytes) 000007 1762490190.964063 write 0x01dba6 thru 0x02b293 (0xd6ee bytes) 000011 1762490190.998409 trunc from 0x040000 to 0x02d527 (0x12ad8 bytes) 000013 1762490191.046695 read 0x0211ff thru 0x02d527 (0xc329 bytes) 000014 1762490191.066698 read 0x000256 thru 0x006962 (0x670d bytes) 000015 1762490191.067286 mapread 0x0253aa thru 0x028ad6 (0x372d bytes) 000017 1762490191.075181 mapread 0x008072 thru 0x01767e (0xf60d bytes) 000022 1762490191.083006 mapwrite 0x010bc7 thru 0x020472 (0xf8ac bytes) 000023 1762490191.150766 read 0x000ee8 thru 0x002d51 (0x1e6a bytes) 000024 1762490191.152315 read 0x00b4a6 thru 0x01b0c4 (0xfc1f bytes) 000025 1762490191.152894 read 0x0006d4 thru 0x00ef91 (0xe8be bytes) 000026 1762490191.153447 read 0x00e9db thru 0x01cf0a (0xe530 bytes) 000027 1762490191.153961 mapread 0x02a1fd thru 0x02d527 (0x332b bytes) 000031 1762490191.156101 trunc from 0x02d528 to 0x004594 (0x28f93 bytes) 000033 1762490191.192315 read 0x000b62 thru 0x004594 (0x3a33 bytes) 000034 1762490191.193686 mapread 0x000414 thru 0x004594 (0x4181 bytes) 000035 1762490191.194527 write 0x00b51b thru 0x00c93b (0x1421 bytes) 000036 1762490191.195427 mapread 0x006116 thru 0x00c93b (0x6826 bytes) 000037 1762490191.207128 mapwrite 0x03336c thru 0x03d5bb (0xa250 bytes) 000038 1762490191.339816 mapread 0x02b40d thru 0x034372 (0x8f66 bytes) 000039 1762490191.340718 mapwrite 0x024bd9 thru 0x033ca5 (0xf0cd bytes) 000043 1762490191.398843 read 0x0377a0 thru 0x03bee1 (0x4742 bytes) 000045 1762490191.400477 write 0x0332a4 thru 0x038f10 (0x5c6d bytes) 000046 1762490191.401450 trunc from 0x03d5bc to 0x0101fb (0x2d3c0 bytes) 000047 1762490191.424642 write 0x01184e thru 0x01d2fa (0xbaad bytes) 000048 1762490191.426240 write 0x020cbb thru 0x02a251 (0x9597 bytes) 000050 1762490191.427859 mapread 0x003e19 thru 0x00da8d (0x9c75 bytes) 000052 1762490191.429399 trunc from 0x02a252 to 0x00c744 (0x1db0d bytes) 000053 1762490191.450317 read 0x0026f8 thru 0x0093d4 (0x6cdd bytes) 000055 1762490191.451331 read 0x007ba9 thru 0x00b428 (0x3880 bytes) 000056 1762490191.451839 read 0x00157d thru 0x005dec (0x4870 bytes) 000057 1762490191.452412 read 0x004b37 thru 0x006a71 (0x1f3b bytes) 000058 1762490191.452903 mapwrite 0x0053ed thru 0x0072c7 (0x1edb bytes) 000059 1762490191.515180 write 0x03b630 thru 0x03ffff (0x49d0 bytes) 000061 1762490191.517028 write 0x0159e6 thru 0x01d0ef (0x770a bytes) 000062 1762490191.527982 read 0x0141da thru 0x0234a4 (0xf2cb bytes) 000063 1762490191.528594 trunc from 0x040000 to 0x02d206 (0x12df9 bytes) 000064 1762490191.547787 trunc from 0x02d207 to 0x022d53 (0xa4b3 bytes) 000065 1762490191.572596 mapread 0x019c15 thru 0x021460 (0x784c bytes) 000066 1762490191.573549 write 0x029d99 thru 0x02e369 (0x45d1 bytes) 000067 1762490191.574602 mapread 0x01f5f3 thru 0x0242b5 (0x4cc3 bytes) 000068 1762490191.588646 mapread 0x02a61e thru 0x02e369 (0x3d4c bytes) 000069 1762490191.589526 write 0x037c5b thru 0x03ffff (0x83a5 bytes) 000070 1762490191.590824 write 0x03064f thru 0x038ce5 (0x8697 bytes) 000071 1762490191.598387 mapwrite 0x0339cf thru 0x039879 (0x5eab bytes) 000072 1762490191.660539 mapread 0x02b50a thru 0x02e962 (0x3459 bytes) 000073 1762490191.661973 trunc from 0x040000 to 0x02265b (0x1d9a4 bytes) 000074 1762490191.687198 write 0x026d2e thru 0x02a793 (0x3a66 bytes) 000075 1762490191.689871 write 0x0341d5 thru 0x03af67 (0x6d93 bytes) 000076 1762490191.691828 trunc from 0x03af68 to 0x00b8d1 (0x2f696 bytes) 000077 1762490191.716378 write 0x03f62f thru 0x03fff2 (0x9c4 bytes) 000078 1762490191.717345 write 0x03eed8 thru 0x03ffff (0x1128 bytes) 000081 1762490191.730174 mapread 0x01418c thru 0x016a51 (0x28c6 bytes) 000084 1762490191.752765 write 0x00337c thru 0x0040bd (0xd42 bytes) 000085 1762490191.765017 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3e90a 000086 1762490192.450162 trunc from 0x040000 to 0x03e909 (0x16f6 bytes) 000088 1762490192.605798 mapread 0x03b888 thru 0x03e909 (0x3082 bytes) 000089 1762490192.630901 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000093 1762490192.843962 trunc from 0x03e90a to 0x0139d0 (0x2af39 bytes) 000094 1762490192.863716 mapwrite 0x02f3ac thru 0x03bebe (0xcb13 bytes) 000095 1762490192.991659 MIRROR_EXTEND to 2 mirrors 000096 1762490193.752337 trunc from 0x03bebf to 0x004a3e (0x37480 bytes) 000098 1762490193.893685 MIRROR_RESYNC 2 mirrors 000099 1762490194.675983 mapread 0x00112d thru 0x004a3e (0x3912 bytes) 000100 1762490194.693262 read 0x000bce thru 0x004a3e (0x3e71 bytes) 000101 1762490194.697346 write 0x00aef6 thru 0x00d2ac (0x23b7 bytes) 000102 1762490194.758407 mapwrite 0x00d9e1 thru 0x01d64f (0xfc6f bytes) 000106 1762490194.904622 trunc from 0x01d650 to 0x023d39 (0x66ea bytes) 000108 1762490194.935847 write 0x03b394 thru 0x03ffff (0x4c6c bytes) 000110 1762490194.938791 mapwrite 0x02d929 thru 0x033284 (0x595c bytes) 000112 1762490195.018365 mapread 0x00eb58 thru 0x01e06c (0xf515 bytes) 000115 1762490195.034299 write 0x012a41 thru 0x0187b5 (0x5d75 bytes) 000118 1762490195.037032 write 0x02ad06 thru 0x035ebe (0xb1b9 bytes) 000122 1762490195.039586 read 0x03d453 thru 0x03ffff (0x2bad bytes) 000123 1762490195.040610 mapread 0x015ca8 thru 0x021391 (0xb6ea bytes) 000124 1762490195.042172 trunc from 0x040000 to 0x03364e (0xc9b1 bytes) 000125 1762490195.062956 trunc from 0x03364f to 0x00d2fc (0x26352 bytes) 000126 1762490195.086581 trunc from 0x00d2fd to 0x0360a3 (0x28da7 bytes) 000127 1762490195.123187 trunc from 0x0360a4 to 0x008efa (0x2d1a9 bytes) 000128 1762490195.153517 read 0x00196c thru 0x008efa (0x758f bytes) 000130 1762490195.167778 read 0x001057 thru 0x001612 (0x5bc bytes) 000131 1762490195.173356 MIRROR_EXTEND to 3 mirrors 000132 1762490195.762126 mapwrite 0x00f958 thru 0x018bc5 (0x926e bytes) 000134 1762490195.948288 mapwrite 0x0048f2 thru 0x014288 (0xf997 bytes) 000135 1762490196.022548 mapwrite 0x009c6b thru 0x015496 (0xb82c bytes) 000138 1762490196.077140 mapwrite 0x02223d thru 0x031db5 (0xfb79 bytes) 000141 1762490196.228463 trunc from 0x031db6 to 0x00e9af (0x23406 bytes) 000143 1762490196.248505 mapwrite 0x022021 thru 0x025d80 (0x3d60 bytes) 000145 1762490196.327540 mapwrite 0x00a1f6 thru 0x010363 (0x616e bytes) 000146 1762490196.388124 mapwrite 0x017e62 thru 0x0245d3 (0xc772 bytes) 000147 1762490196.455306 trunc from 0x025d81 to 0x010779 (0x15607 bytes) 000148 1762490196.476901 read 0x00089b thru 0x010354 (0xfaba bytes) 000151 1762490196.484484 mapread 0x00554f thru 0x010779 (0xb22b bytes) 000152 1762490196.486466 write 0x02d5ae thru 0x0358b8 (0x830b bytes) 000155 1762490196.498911 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000158 1762490196.701528 mapwrite 0x02a892 thru 0x037d5a (0xd4c9 bytes) 000159 1762490196.798939 write 0x003a13 thru 0x00bd1f (0x830d bytes) 000160 1762490196.812865 write 0x0257cf thru 0x028272 (0x2aa4 bytes) 000161 1762490196.813711 mapwrite 0x00779a thru 0x008c7e (0x14e5 bytes) 000163 1762490196.862174 write 0x0380ba thru 0x03ffff (0x7f46 bytes) 000164 1762490196.863339 write 0x02e37d thru 0x034195 (0x5e19 bytes) 000168 1762490196.866053 mapread 0x019f34 thru 0x01a05d (0x12a bytes) 000171 1762490196.881566 mapread 0x0079e0 thru 0x014406 (0xca27 bytes) 000172 1762490196.897635 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000175 1762490197.093133 trunc from 0x040000 to 0x02371a (0x1c8e5 bytes) 000176 1762490197.122959 mapwrite 0x020446 thru 0x02d825 (0xd3e0 bytes) 000179 1762490197.247502 MIRROR_EXTEND to 2 mirrors 000182 1762490197.928809 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000183 1762490198.081185 mapwrite 0x0071f4 thru 0x015175 (0xdf82 bytes) 000185 1762490198.195323 read 0x01a247 thru 0x027c75 (0xda2f bytes) 000186 1762490198.208469 MIRROR_EXTEND to 2 mirrors 000187 1762490198.738858 write 0x01bd3b thru 0x022192 (0x6458 bytes) 000189 1762490198.843914 write 0x01d08f thru 0x021684 (0x45f6 bytes) 000191 1762490198.853913 MIRROR_EXTEND to 3 mirrors 000192 1762490199.605999 MIRROR_RESYNC 3 mirrors 000193 1762490200.359013 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000194 1762490200.829367 trunc from 0x02d826 to 0x02a902 (0x2f23 bytes) 000195 1762490200.934250 mapwrite 0x002459 thru 0x00c194 (0x9d3c bytes) 000196 1762490201.044546 mapread 0x0202f9 thru 0x02a902 (0xa60a bytes) 000197 1762490201.060357 trunc from 0x02a903 to 0x019449 (0x114b9 bytes) 000199 1762490201.084322 trunc from 0x01944a to 0x016e7d (0x25cc bytes) 000200 1762490201.104288 mapwrite 0x00014a thru 0x00ddb6 (0xdc6d bytes) 000201 1762490201.165543 mapread 0x0081a1 thru 0x014e20 (0xcc80 bytes) 000202 1762490201.175444 MIRROR_RESYNC 2 mirrors 000203 1762490201.680650 mapread 0x00fbc1 thru 0x016e7d (0x72bd bytes) 000205 1762490201.710271 write 0x007492 thru 0x008b0d (0x167c bytes) 000206 1762490201.797928 mapread 0x010bb8 thru 0x01158f (0x9d8 bytes) 000207 1762490201.814459 MIRROR_EXTEND to 3 mirrors 000208 1762490202.463672 read 0x006f63 thru 0x01218c (0xb22a bytes) 000209 1762490202.479897 mapwrite 0x02acf6 thru 0x03a237 (0xf542 bytes) 000210 1762490202.660803 read 0x02f05e thru 0x03a237 (0xb1da bytes) 000212 1762490202.661877 trunc from 0x03a238 to 0x01770e (0x22b29 bytes) 000213 1762490202.692447 MIRROR_RESYNC 3 mirrors 000214 1762490203.396689 mapwrite 0x00069d thru 0x00b0a1 (0xaa05 bytes) 000215 1762490203.607279 write 0x005156 thru 0x00633d (0x11e8 bytes) 000216 1762490203.608152 trunc from 0x01770f to 0x033bbf (0x1c4b1 bytes) 000217 1762490203.649918 mapread 0x004705 thru 0x0092ab (0x4ba7 bytes) 000218 1762490203.650774 read 0x028f29 thru 0x033bbf (0xac97 bytes) 000221 1762490203.663641 read 0x025fa5 thru 0x031876 (0xb8d2 bytes) 000223 1762490203.682989 trunc from 0x033bc0 to 0x01b1aa (0x18a15 bytes) 000224 1762490203.700645 mapwrite 0x0283db thru 0x034621 (0xc247 bytes) 000225 1762490203.839503 mapwrite 0x01e5b0 thru 0x022bf4 (0x4645 bytes) 000227 1762490203.919185 mapread 0x023f56 thru 0x02dd7b (0x9e26 bytes) 000229 1762490203.958702 mapwrite 0x034827 thru 0x03fa0a (0xb1e4 bytes) 000230 1762490204.035370 mapread 0x02aa43 thru 0x03680f (0xbdcd bytes) 000232 1762490204.037816 write 0x0313ec thru 0x034a93 (0x36a8 bytes) 000234 1762490204.040312 trunc from 0x03fa0b to 0x01c8c1 (0x23149 bytes) 000235 1762490204.061364 write 0x016e59 thru 0x01ffe1 (0x9189 bytes) 000237 1762490204.069231 MIRROR_RESYNC 3 mirrors 000238 1762490204.735939 mapwrite 0x033417 thru 0x03fb6c (0xc756 bytes) 000239 1762490204.891965 mapwrite 0x01c442 thru 0x027eb2 (0xba71 bytes) 000240 1762490204.958399 mapread 0x038d07 thru 0x03b398 (0x2692 bytes) 000241 1762490204.960318 write 0x0205c5 thru 0x0304f9 (0xff35 bytes) 000242 1762490204.961636 write 0x0390a8 thru 0x03ffff (0x6f58 bytes) 000243 1762490204.962618 mapread 0x018c1b thru 0x026340 (0xd726 bytes) 000244 1762490204.968949 MIRROR_EXTEND to 4 mirrors 000245 1762490205.656702 trunc from 0x040000 to 0x024562 (0x1ba9d bytes) 000246 1762490205.727501 write 0x02de4a thru 0x03095b (0x2b12 bytes) 000247 1762490205.728599 mapwrite 0x02d315 thru 0x036b3a (0x9826 bytes) 000250 1762490205.855453 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000251 1762490206.139657 read 0x036899 thru 0x036b3a (0x2a2 bytes) 000252 1762490206.148825 trunc from 0x036b3b to 0x022019 (0x14b21 bytes) 000254 1762490206.170097 mapwrite 0x001649 thru 0x00d092 (0xba4a bytes) 000257 1762490206.247447 mapread 0x01f82c thru 0x022019 (0x27ee bytes) 000258 1762490206.256878 read 0x01de89 thru 0x022019 (0x4191 bytes) 000261 1762490206.258416 read 0x01838b thru 0x01b8ae (0x3524 bytes) 000263 1762490206.259396 write 0x00e1c2 thru 0x00e4de (0x31d bytes) 000268 1762490206.262278 trunc from 0x02201a to 0x018d22 (0x92f7 bytes) 000269 1762490206.289325 trunc from 0x018d23 to 0x012077 (0x6cab bytes) 000272 1762490206.316021 write 0x005935 thru 0x00a282 (0x494e bytes) 000276 1762490206.318269 mapwrite 0x0169d6 thru 0x01d382 (0x69ad bytes) 000277 1762490206.412327 mapread 0x002410 thru 0x00dcdb (0xb8cc bytes) 000278 1762490206.413284 mapread 0x0128e8 thru 0x016b53 (0x426c bytes) 000279 1762490206.414006 trunc from 0x01d383 to 0x0366a9 (0x19327 bytes) 000280 1762490206.434473 mapwrite 0x03fb28 thru 0x03ffff (0x4d8 bytes) 000281 1762490206.525403 MIRROR_RESYNC 3 mirrors 000284 1762490207.217656 MIRROR_RESYNC 3 mirrors 000286 1762490207.543479 mapread 0x00857c thru 0x011047 (0x8acc bytes) 000288 1762490207.561220 read 0x0284cf thru 0x031dab (0x98dd bytes) 000289 1762490207.573320 read 0x0046f7 thru 0x004914 (0x21e bytes) 000290 1762490207.579283 MIRROR_RESYNC 3 mirrors 000291 1762490207.876105 read 0x004275 thru 0x010758 (0xc4e4 bytes) 000292 1762490207.878368 mapread 0x011ac7 thru 0x016a3b (0x4f75 bytes) 000295 1762490207.895112 mapread 0x024e66 thru 0x0282e5 (0x3480 bytes) 000297 1762490207.918632 mapread 0x02456b thru 0x02ee54 (0xa8ea bytes) truncating to largest ever: 0x3f891 000300 1762490207.921320 trunc from 0x040000 to 0x03f890 (0x76f bytes) 000301 1762490208.051501 mapread 0x039cf0 thru 0x03f890 (0x5ba1 bytes) 000303 1762490208.062042 write 0x029217 thru 0x02b181 (0x1f6b bytes) 000305 1762490208.073319 write 0x031e7d thru 0x0335df (0x1763 bytes) 000306 1762490208.078052 MIRROR_EXTEND to 4 mirrors 000308 1762490208.615278 mapread 0x037cc9 thru 0x03e5e0 (0x6918 bytes) 000309 1762490208.651744 trunc from 0x03f891 to 0x01f782 (0x2010e bytes) 000310 1762490208.788837 write 0x0014a8 thru 0x006f79 (0x5ad2 bytes) 000311 1762490208.797913 read 0x005860 thru 0x00d020 (0x77c1 bytes) 000312 1762490208.805450 mapread 0x018abf thru 0x01f782 (0x6cc4 bytes) 000313 1762490208.815954 trunc from 0x01f783 to 0x00098d (0x1edf5 bytes) 000315 1762490208.856706 MIRROR_EXTEND to 5 mirrors 000316 1762490209.700371 MIRROR_EXTEND to 6 mirrors 000317 1762490210.446745 mapread 0x00006a thru 0x00098d (0x924 bytes) 000318 1762490210.456564 read 0x0006bc thru 0x00098d (0x2d2 bytes) 000319 1762490210.456966 mapread 0x00032c thru 0x00098d (0x662 bytes) 000320 1762490210.457509 mapwrite 0x02c50d thru 0x02ce45 (0x939 bytes) 000321 1762490210.610040 trunc from 0x02ce46 to 0x01de34 (0xf011 bytes) 000326 1762490210.635979 trunc from 0x01de35 to 0x0060ef (0x17d45 bytes) 000329 1762490210.667706 MIRROR_RESYNC 6 mirrors 000331 1762490212.065823 write 0x028265 thru 0x02f3e2 (0x717e bytes) 000335 1762490212.136984 trunc from 0x02f3e3 to 0x033dd3 (0x49f1 bytes) 000336 1762490212.193223 write 0x020dba thru 0x02a0fa (0x9341 bytes) 000337 1762490212.200540 read 0x004fd1 thru 0x006aaf (0x1adf bytes) 000338 1762490212.207389 mapwrite 0x00f590 thru 0x01094d (0x13be bytes) 000339 1762490212.317583 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000341 1762490212.639388 MIRROR_EXTEND to 6 mirrors 000343 1762490213.634801 mapread 0x013bbd thru 0x018bab (0x4fef bytes) 000345 1762490213.696617 trunc from 0x033dd4 to 0x022cd0 (0x11103 bytes) 000346 1762490213.836259 mapread 0x00eb33 thru 0x01ab4c (0xc01a bytes) 000347 1762490213.869430 MIRROR_RESYNC 6 mirrors 000348 1762490215.106072 mapread 0x01afc4 thru 0x01b6ce (0x70b bytes) 000349 1762490215.117277 mapwrite 0x01466a thru 0x018089 (0x3a20 bytes) 000350 1762490215.298223 mapwrite 0x03510c thru 0x03bc57 (0x6b4c bytes) 000355 1762490215.375861 read 0x016a55 thru 0x01c53c (0x5ae8 bytes) 000356 1762490215.377278 write 0x009744 thru 0x010be5 (0x74a2 bytes) 000357 1762490215.386180 mapwrite 0x029ca8 thru 0x031b18 (0x7e71 bytes) 000359 1762490215.441805 write 0x01f730 thru 0x02ef76 (0xf847 bytes) 000361 1762490215.457936 MIRROR_EXTEND to 7 mirrors 000363 1762490216.198185 trunc from 0x03bc58 to 0x02cc85 (0xefd2 bytes) 000364 1762490216.392658 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000365 1762490216.635491 mapread 0x0259fa thru 0x02b174 (0x577b bytes) 000366 1762490216.676224 write 0x0084fe thru 0x0122bc (0x9dbf bytes) 000368 1762490216.690959 trunc from 0x02cc86 to 0x017f89 (0x14cfc bytes) 000369 1762490216.715379 read 0x0076d2 thru 0x016d5a (0xf689 bytes) 000372 1762490216.723649 read 0x00ede1 thru 0x017f89 (0x91a9 bytes) 000373 1762490216.733984 mapwrite 0x022c83 thru 0x026e3b (0x41b9 bytes) 000374 1762490216.870687 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000375 1762490217.174929 trunc from 0x026e3c to 0x03a21a (0x133df bytes) 000377 1762490217.204306 mapwrite 0x024c0a thru 0x034384 (0xf77b bytes) 000378 1762490217.285673 write 0x029cc6 thru 0x02c42b (0x2766 bytes) 000380 1762490217.287158 read 0x00492c thru 0x00d077 (0x874c bytes) 000381 1762490217.301826 MIRROR_EXTEND to 6 mirrors 000382 1762490218.097818 MIRROR_EXTEND to 7 mirrors 000383 1762490218.767417 write 0x00cba2 thru 0x00f445 (0x28a4 bytes) 000386 1762490218.849843 mapread 0x021f75 thru 0x02cdfd (0xae89 bytes) 000387 1762490218.867901 mapread 0x01c02c thru 0x0203b9 (0x438e bytes) 000389 1762490218.869407 mapwrite 0x00b61f thru 0x00ff8e (0x4970 bytes) 000392 1762490218.945468 mapwrite 0x039508 thru 0x03e79e (0x5297 bytes) 000393 1762490219.057860 read 0x023bae thru 0x02ed7b (0xb1ce bytes) 000394 1762490219.058702 trunc from 0x03e79f to 0x037bdb (0x6bc3 bytes) 000396 1762490219.091768 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000397 1762490219.324785 read 0x015567 thru 0x01beb9 (0x6953 bytes) 000399 1762490219.332828 mapwrite 0x00a52b thru 0x011b4a (0x7620 bytes) 000402 1762490219.394737 mapread 0x027ced thru 0x02a54e (0x2862 bytes) 000404 1762490219.405858 read 0x014159 thru 0x01a454 (0x62fc bytes) 000410 1762490219.413521 trunc from 0x037bdc to 0x0329eb (0x51f0 bytes) 000413 1762490219.433359 trunc from 0x0329ec to 0x03ebb6 (0xc1cb bytes) 000414 1762490219.468657 read 0x00b702 thru 0x012cc6 (0x75c5 bytes) 000417 1762490219.471705 trunc from 0x03ebb7 to 0x015b19 (0x2909d bytes) 000418 1762490219.493551 trunc from 0x015b1a to 0x03734d (0x21834 bytes) 000420 1762490219.511246 mapread 0x01136c thru 0x016269 (0x4efe bytes) 000421 1762490219.530270 read 0x0222ef thru 0x023a3a (0x174c bytes) 000422 1762490219.532023 mapwrite 0x03176c thru 0x03621c (0x4ab1 bytes) 000423 1762490219.584444 mapread 0x0095f7 thru 0x00e80e (0x5218 bytes) 000424 1762490219.585770 read 0x017559 thru 0x0187f5 (0x129d bytes) 000427 1762490219.588253 trunc from 0x03734e to 0x00cb44 (0x2a809 bytes) 000428 1762490219.607458 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000430 1762490219.878233 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000431 1762490220.134726 mapwrite 0x024ff7 thru 0x027743 (0x274d bytes) 000432 1762490220.307361 mapwrite 0x0001cc thru 0x007a92 (0x78c7 bytes) 000433 1762490220.412457 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000434 1762490220.720129 write 0x0005ae thru 0x00c1a0 (0xbbf3 bytes) 000435 1762490220.736313 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000436 1762490220.963664 read 0x004401 thru 0x00831d (0x3f1d bytes) 000437 1762490220.977459 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000438 1762490221.134614 trunc from 0x027744 to 0x03d70f (0x15fcc bytes) 000440 1762490221.160497 trunc from 0x03d710 to 0x015735 (0x27fda bytes) 000441 1762490221.197249 MIRROR_EXTEND to 2 mirrors 000444 1762490221.825600 mapread 0x012633 thru 0x015735 (0x3103 bytes) 000447 1762490221.844304 mapread 0x002cdb thru 0x006f39 (0x425f bytes) 000448 1762490221.854402 mapread 0x011f1a thru 0x015735 (0x381c bytes) 000451 1762490221.856438 mapread 0x005f35 thru 0x011c17 (0xbce3 bytes) 000452 1762490221.857282 write 0x020947 thru 0x022602 (0x1cbc bytes) 000454 1762490221.930923 read 0x00691e thru 0x0150c1 (0xe7a4 bytes) 000456 1762490221.964582 MIRROR_EXTEND to 3 mirrors 000459 1762490222.918549 trunc from 0x022603 to 0x02bdab (0x97a9 bytes) 000461 1762490223.018536 read 0x021e18 thru 0x026f61 (0x514a bytes) 000462 1762490223.037237 MIRROR_RESYNC 3 mirrors 000464 1762490223.811235 write 0x036e7a thru 0x03df60 (0x70e7 bytes) 000468 1762490223.876520 mapwrite 0x0267ae thru 0x0314eb (0xad3e bytes) 000469 1762490224.006302 mapwrite 0x00c03c thru 0x00d6eb (0x16b0 bytes) 000470 1762490224.128733 mapwrite 0x012a9f thru 0x0200d1 (0xd633 bytes) 000471 1762490224.194066 mapread 0x011b95 thru 0x017663 (0x5acf bytes) 000473 1762490224.201298 MIRROR_RESYNC 3 mirrors 000475 1762490224.902863 mapwrite 0x00f95d thru 0x012e77 (0x351b bytes) 000476 1762490225.119150 trunc from 0x03df61 to 0x02b06b (0x12ef5 bytes) 000477 1762490225.208274 mapread 0x0007e3 thru 0x00538c (0x4baa bytes) 000478 1762490225.226705 read 0x00a44b thru 0x011cf8 (0x78ae bytes) 000479 1762490225.227454 trunc from 0x02b06c to 0x02d705 (0x269a bytes) 000481 1762490225.250392 read 0x027587 thru 0x02d705 (0x617f bytes) 000482 1762490225.266732 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000483 1762490225.512361 read 0x02b411 thru 0x02d705 (0x22f5 bytes) 000484 1762490225.518346 write 0x027980 thru 0x030928 (0x8fa9 bytes) 000486 1762490225.532706 MIRROR_RESYNC 2 mirrors 000489 1762490226.279918 trunc from 0x030929 to 0x02509e (0xb88a bytes) 000493 1762490226.427449 write 0x03d86c thru 0x03ffff (0x2794 bytes) 000495 1762490226.430440 read 0x03eee2 thru 0x03ffff (0x111e bytes) 000496 1762490226.444334 MIRROR_RESYNC 2 mirrors 000497 1762490227.092878 trunc from 0x040000 to 0x01dd79 (0x22286 bytes) 000498 1762490227.264930 mapwrite 0x01fb54 thru 0x025059 (0x5506 bytes) 000499 1762490227.417612 trunc from 0x02505a to 0x016c7b (0xe3de bytes) 000500 1762490227.437916 mapwrite 0x038336 thru 0x03e93f (0x660a bytes) 000502 1762490227.514462 write 0x01b38f thru 0x026936 (0xb5a8 bytes) 000503 1762490227.533826 write 0x02b5a8 thru 0x03a4bc (0xef15 bytes) 000504 1762490227.549425 MIRROR_RESYNC 2 mirrors 000505 1762490228.280501 read 0x03e300 thru 0x03e93f (0x640 bytes) 000506 1762490228.290948 mapwrite 0x018590 thru 0x01e781 (0x61f2 bytes) 000507 1762490228.443471 trunc from 0x03e940 to 0x006d0f (0x37c30 bytes) 000508 1762490228.512128 mapwrite 0x03d1d1 thru 0x03ffff (0x2e2f bytes) 000509 1762490228.585275 read 0x003347 thru 0x007791 (0x444b bytes) 000510 1762490228.601530 read 0x00f1bf thru 0x014eca (0x5d0c bytes) 000512 1762490228.602723 mapread 0x01dcde thru 0x02897a (0xac9d bytes) 000513 1762490228.608008 MIRROR_RESYNC 2 mirrors 000514 1762490229.344573 read 0x022a35 thru 0x02d1a3 (0xa76f bytes) 000515 1762490229.356459 write 0x000283 thru 0x00d733 (0xd4b1 bytes) 000516 1762490229.459962 mapread 0x031d9e thru 0x032daf (0x1012 bytes) 000517 1762490229.490021 mapwrite 0x01a1fc thru 0x01c2ea (0x20ef bytes) 000519 1762490229.583339 read 0x0382d8 thru 0x03ffff (0x7d28 bytes) 000520 1762490229.595879 trunc from 0x040000 to 0x01ef10 (0x210ef bytes) 000521 1762490229.702934 read 0x003fb0 thru 0x00cfb3 (0x9004 bytes) 000522 1762490229.704577 trunc from 0x01ef11 to 0x025d22 (0x6e12 bytes) 000523 1762490229.731300 read 0x01571c thru 0x02143f (0xbd24 bytes) 000524 1762490229.745564 write 0x0107be thru 0x015d70 (0x55b3 bytes) 000525 1762490229.748479 write 0x01e41d thru 0x0266fb (0x82df bytes) 000526 1762490229.749609 write 0x00dc7b thru 0x01326b (0x55f1 bytes) 000529 1762490229.754067 write 0x010f05 thru 0x0189e3 (0x7adf bytes) 000531 1762490229.756936 trunc from 0x0266fc to 0x039016 (0x1291b bytes) 000533 1762490229.793902 trunc from 0x039017 to 0x029717 (0xf8ff bytes) 000534 1762490229.822111 MIRROR_EXTEND to 3 mirrors 000538 1762490230.482686 trunc from 0x029718 to 0x007042 (0x226d5 bytes) 000541 1762490230.575261 mapwrite 0x0167c9 thru 0x025a65 (0xf29d bytes) 000542 1762490230.721506 MIRROR_RESYNC 3 mirrors 000543 1762490231.774312 MIRROR_RESYNC 3 mirrors 000545 1762490232.097922 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000547 1762490232.533424 MIRROR_EXTEND to 3 mirrors 000548 1762490233.334242 mapwrite 0x019654 thru 0x021474 (0x7e21 bytes) 000549 1762490233.550012 mapread 0x00cccf thru 0x00f928 (0x2c5a bytes) 000550 1762490233.565394 write 0x036ca9 thru 0x03d16a (0x64c2 bytes) 000551 1762490233.576826 MIRROR_RESYNC 3 mirrors 000553 1762490234.553673 read 0x013eb1 thru 0x01c6a4 (0x87f4 bytes) 000554 1762490234.577263 MIRROR_EXTEND to 4 mirrors 000557 1762490235.518366 trunc from 0x03d16b to 0x00223e (0x3af2c bytes) 000559 1762490235.628538 read 0x00102d thru 0x00223e (0x1212 bytes) 000560 1762490235.638343 mapread 0x001069 thru 0x00223e (0x11d6 bytes) 000561 1762490235.640000 read 0x000d0f thru 0x00223e (0x1530 bytes) 000565 1762490235.649693 write 0x006d04 thru 0x00c55d (0x585a bytes) 000566 1762490235.651715 trunc from 0x00c55e to 0x00d956 (0x13f9 bytes) 000569 1762490235.683600 trunc from 0x00d957 to 0x02d824 (0x1fece bytes) 000570 1762490235.708818 mapwrite 0x03cef6 thru 0x03f85e (0x2969 bytes) 000571 1762490235.869295 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000572 1762490236.103950 mapread 0x02d5f3 thru 0x03920d (0xbc1b bytes) 000573 1762490236.140736 read 0x02246e thru 0x03176d (0xf300 bytes) 000576 1762490236.156335 MIRROR_RESYNC 3 mirrors 000577 1762490237.189604 MIRROR_RESYNC 3 mirrors 000578 1762490237.650382 mapread 0x0348b4 thru 0x03b56d (0x6cba bytes) 000579 1762490237.673292 trunc from 0x03f85f to 0x006c41 (0x38c1d bytes) 000581 1762490237.737797 mapwrite 0x03ed13 thru 0x03ffff (0x12ed bytes) 000582 1762490237.838825 read 0x01d650 thru 0x025c9b (0x864c bytes) 000584 1762490237.850831 read 0x02fc3f thru 0x038809 (0x8bcb bytes) 000585 1762490237.871035 mapwrite 0x03f347 thru 0x03ffff (0xcb9 bytes) 000588 1762490237.930699 MIRROR_RESYNC 3 mirrors 000589 1762490238.670323 trunc from 0x040000 to 0x029781 (0x1687e bytes) 000590 1762490238.763950 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000592 1762490239.015324 read 0x0101e1 thru 0x013c41 (0x3a61 bytes) 000593 1762490239.023046 mapwrite 0x004f91 thru 0x00e7b4 (0x9824 bytes) 000594 1762490239.083827 mapwrite 0x0051c3 thru 0x009e2a (0x4c68 bytes) 000596 1762490239.125943 mapread 0x01ba67 thru 0x01d76e (0x1d08 bytes) 000598 1762490239.143339 write 0x029b8a thru 0x02e134 (0x45ab bytes) 000599 1762490239.154302 read 0x0160f2 thru 0x02169b (0xb5aa bytes) 000600 1762490239.156958 trunc from 0x02e135 to 0x033652 (0x551e bytes) 000604 1762490239.181618 read 0x002354 thru 0x005503 (0x31b0 bytes) 000605 1762490239.182349 read 0x019723 thru 0x01a2f5 (0xbd3 bytes) 000606 1762490239.183449 mapwrite 0x0114aa thru 0x012deb (0x1942 bytes) 000607 1762490239.228144 trunc from 0x033653 to 0x031e47 (0x180b bytes) 000609 1762490239.248816 mapwrite 0x013779 thru 0x02370b (0xff93 bytes) 000614 1762490239.303527 mapread 0x0225c1 thru 0x02d0fb (0xab3b bytes) 000615 1762490239.305390 read 0x007e64 thru 0x008ac5 (0xc62 bytes) 000616 1762490239.306891 trunc from 0x031e48 to 0x02f004 (0x2e43 bytes) 000617 1762490239.330000 mapwrite 0x00367c thru 0x00ed9d (0xb722 bytes) 000619 1762490239.385463 mapwrite 0x026503 thru 0x028bc0 (0x26be bytes) 000620 1762490239.424661 MIRROR_EXTEND to 3 mirrors 000622 1762490239.953423 mapwrite 0x03a37c thru 0x03ffff (0x5c84 bytes) 000623 1762490240.159485 mapwrite 0x009658 thru 0x00ce86 (0x382f bytes) 000624 1762490240.222359 read 0x000b76 thru 0x010ab0 (0xff3b bytes) 000625 1762490240.230585 mapwrite 0x01e8c5 thru 0x01fa76 (0x11b2 bytes) 000626 1762490240.298952 mapread 0x000aa2 thru 0x0050a5 (0x4604 bytes) 000628 1762490240.300231 read 0x039873 thru 0x03ffff (0x678d bytes) 000629 1762490240.310602 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000631 1762490240.628248 read 0x01d08c thru 0x021de3 (0x4d58 bytes) 000632 1762490240.652734 MIRROR_RESYNC 2 mirrors 000633 1762490241.372311 MIRROR_RESYNC 2 mirrors 000634 1762490241.560053 write 0x01dc41 thru 0x02be72 (0xe232 bytes) 000635 1762490241.648617 mapread 0x038db3 thru 0x03ffff (0x724d bytes) 000636 1762490241.657733 trunc from 0x040000 to 0x00f190 (0x30e6f bytes) 000638 1762490241.698786 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000639 1762490241.878542 read 0x000c00 thru 0x001057 (0x458 bytes) 000640 1762490241.883899 write 0x03075d thru 0x0328ec (0x2190 bytes) 000641 1762490241.884932 trunc from 0x0328ed to 0x030220 (0x26cc bytes) 000642 1762490241.905568 read 0x01ed93 thru 0x02a823 (0xba91 bytes) 000643 1762490241.918279 trunc from 0x030221 to 0x02f1fa (0x1026 bytes) 000644 1762490241.937005 write 0x01f295 thru 0x021e0b (0x2b77 bytes) 000647 1762490241.938904 mapwrite 0x018ddf thru 0x01b5e3 (0x2805 bytes) 000648 1762490241.995332 write 0x00e7f1 thru 0x012cca (0x44da bytes) 000649 1762490241.998010 mapwrite 0x02c64f thru 0x03c3a5 (0xfd57 bytes) 000650 1762490242.107612 trunc from 0x03c3a6 to 0x0201ee (0x1c1b7 bytes) 000653 1762490242.124201 mapread 0x00d769 thru 0x01260a (0x4ea2 bytes) 000655 1762490242.129936 trunc from 0x0201ef to 0x03168b (0x1149d bytes) 000657 1762490242.152867 write 0x007dea thru 0x008039 (0x250 bytes) 000658 1762490242.155596 read 0x0300ae thru 0x03168b (0x15de bytes) 000659 1762490242.167340 mapwrite 0x03cb4e thru 0x03ffff (0x34b2 bytes) 000660 1762490242.238134 read 0x039a48 thru 0x03ffff (0x65b8 bytes) 000661 1762490242.238732 mapwrite 0x0397e4 thru 0x03ffff (0x681c bytes) 000664 1762490242.285695 mapread 0x01f2ed thru 0x02b02c (0xbd40 bytes) 000666 1762490242.287620 mapread 0x01a30a thru 0x020ca0 (0x6997 bytes) 000669 1762490242.290383 mapread 0x008f22 thru 0x0106d7 (0x77b6 bytes) 000670 1762490242.292368 read 0x01b6c6 thru 0x027f1c (0xc857 bytes) 000671 1762490242.292936 trunc from 0x040000 to 0x01512b (0x2aed4 bytes) 000672 1762490242.319616 read 0x010999 thru 0x01512b (0x4793 bytes) 000674 1762490242.322550 write 0x00744d thru 0x00bb8a (0x473e bytes) 000675 1762490242.323527 mapwrite 0x00bec9 thru 0x012916 (0x6a4e bytes) 000676 1762490242.372453 mapread 0x010601 thru 0x01512b (0x4b2b bytes) 000681 1762490242.375263 mapwrite 0x03bfc4 thru 0x03ffff (0x403c bytes) 000682 1762490242.489298 read 0x002640 thru 0x00ff12 (0xd8d3 bytes) 000684 1762490242.493666 mapwrite 0x0149b6 thru 0x01674d (0x1d98 bytes) 000685 1762490242.563021 write 0x027cf2 thru 0x02df11 (0x6220 bytes) 000687 1762490242.573772 read 0x025428 thru 0x031c05 (0xc7de bytes) 000689 1762490242.575286 read 0x0290a4 thru 0x02c277 (0x31d4 bytes) 000690 1762490242.575825 mapwrite 0x02e30d thru 0x038f2a (0xac1e bytes) 000691 1762490242.704313 trunc from 0x040000 to 0x006829 (0x397d6 bytes) 000693 1762490242.745061 write 0x02dcd1 thru 0x033be5 (0x5f15 bytes) 000694 1762490242.746340 mapread 0x00829f thru 0x00cbff (0x4961 bytes) 000695 1762490242.758048 mapwrite 0x0192a8 thru 0x028f03 (0xfc5c bytes) 000696 1762490242.827681 mapread 0x01110d thru 0x0175d6 (0x64ca bytes) 000698 1762490242.829688 mapread 0x00aa06 thru 0x0155fa (0xabf5 bytes) 000701 1762490242.831921 write 0x034643 thru 0x03e907 (0xa2c5 bytes) 000702 1762490242.833300 mapread 0x020c04 thru 0x0255dc (0x49d9 bytes) 000703 1762490242.834111 write 0x001c44 thru 0x00e0cd (0xc48a bytes) 000704 1762490242.835185 write 0x021ee6 thru 0x022ada (0xbf5 bytes) 000705 1762490242.836034 read 0x032a47 thru 0x03e907 (0xbec1 bytes) 000706 1762490242.841002 mapread 0x0141a4 thru 0x016fe9 (0x2e46 bytes) 000710 1762490242.843530 mapread 0x000533 thru 0x00d569 (0xd037 bytes) 000712 1762490242.844880 mapwrite 0x02c04f thru 0x02d572 (0x1524 bytes) 000714 1762490242.905627 read 0x0251bd thru 0x028371 (0x31b5 bytes) 000718 1762490242.909400 mapwrite 0x00ae39 thru 0x0140a1 (0x9269 bytes) 000720 1762490242.966893 trunc from 0x03e908 to 0x014e9e (0x29a69 bytes) 000724 1762490242.991408 write 0x00d040 thru 0x00e9bf (0x1980 bytes) 000725 1762490242.992324 trunc from 0x014e9f to 0x00a8f3 (0xa5ab bytes) 000730 1762490243.046334 read 0x000aab thru 0x005805 (0x4d5b bytes) 000731 1762490243.050883 mapwrite 0x002f01 thru 0x012c7b (0xfd7b bytes) 000732 1762490243.256824 mapread 0x006d4a thru 0x012c7b (0xbf32 bytes) 000733 1762490243.262187 mapread 0x0089c8 thru 0x00d9b3 (0x4fec bytes) 000737 1762490243.268420 mapwrite 0x00e9c7 thru 0x0168f2 (0x7f2c bytes) 000738 1762490243.403688 trunc from 0x0168f3 to 0x000bde (0x15d14 bytes) 000739 1762490243.445942 mapwrite 0x0308d9 thru 0x036c88 (0x63b0 bytes) 000740 1762490243.586504 write 0x02d3f3 thru 0x03bb52 (0xe760 bytes) 000742 1762490243.588687 read 0x020251 thru 0x0250d8 (0x4e88 bytes) 000744 1762490243.592378 read 0x031b30 thru 0x03bb52 (0xa023 bytes) 000746 1762490243.595649 trunc from 0x03bb53 to 0x032650 (0x9502 bytes) 000747 1762490243.617681 mapread 0x01b850 thru 0x02513e (0x98ef bytes) 000748 1762490243.618682 trunc from 0x032651 to 0x018287 (0x1a3c9 bytes) 000750 1762490243.651376 read 0x004fbd thru 0x00a6ff (0x5743 bytes) 000752 1762490243.662216 trunc from 0x018288 to 0x00d7fb (0xaa8c bytes) 000753 1762490243.684754 write 0x011e3b thru 0x01ca4e (0xac14 bytes) 000754 1762490243.689437 mapread 0x003032 thru 0x00636b (0x333a bytes) 000755 1762490243.692051 read 0x014712 thru 0x01a726 (0x6015 bytes) 000759 1762490243.694570 mapwrite 0x0168eb thru 0x01d341 (0x6a57 bytes) 000761 1762490243.783718 trunc from 0x01d342 to 0x03c160 (0x1ee1f bytes) 000762 1762490243.807530 mapwrite 0x003f06 thru 0x00e9c9 (0xaac4 bytes) 000763 1762490243.845917 mapread 0x003b57 thru 0x00b110 (0x75ba bytes) 000764 1762490243.847181 trunc from 0x03c161 to 0x003eb7 (0x382a9 bytes) 000765 1762490243.869166 read 0x000678 thru 0x003eb7 (0x3840 bytes) 000766 1762490243.873312 write 0x01048a thru 0x017d61 (0x78d8 bytes) 000767 1762490243.874478 write 0x03b382 thru 0x03ffff (0x4c7e bytes) 000769 1762490243.877757 trunc from 0x040000 to 0x03de8f (0x2170 bytes) 000770 1762490243.903912 mapread 0x02d95e thru 0x036542 (0x8be5 bytes) 000771 1762490243.921860 read 0x019928 thru 0x02701b (0xd6f4 bytes) 000772 1762490243.922786 write 0x01c173 thru 0x022dde (0x6c6c bytes) 000773 1762490243.923780 mapwrite 0x029c33 thru 0x039a33 (0xfe01 bytes) 000775 1762490244.007687 write 0x0132c8 thru 0x013935 (0x66e bytes) 000778 1762490244.009296 write 0x00451d thru 0x00c9dc (0x84c0 bytes) 000784 1762490244.014547 write 0x03634e thru 0x03fc41 (0x98f4 bytes) 000786 1762490244.017200 mapread 0x00d4df thru 0x01172b (0x424d bytes) 000788 1762490244.022284 read 0x001c3d thru 0x00ea47 (0xce0b bytes) 000791 1762490244.024033 read 0x0376e2 thru 0x03d5a0 (0x5ebf bytes) 000792 1762490244.024626 trunc from 0x03fc42 to 0x021346 (0x1e8fb bytes) 000793 1762490244.057474 MIRROR_EXTEND to 2 mirrors 000794 1762490244.837557 mapread 0x018bd3 thru 0x021346 (0x8774 bytes) 000796 1762490244.860418 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000797 1762490245.014659 read 0x014713 thru 0x01ead3 (0xa3c1 bytes) 000802 1762490245.026311 mapread 0x00a3f2 thru 0x01458a (0xa199 bytes) 000805 1762490245.044137 trunc from 0x021347 to 0x017113 (0xa233 bytes) 000806 1762490245.079962 trunc from 0x017114 to 0x00204f (0x150c4 bytes) 000807 1762490245.097795 trunc from 0x002050 to 0x03005d (0x2e00e bytes) 000809 1762490245.128163 read 0x0291f5 thru 0x02bc48 (0x2a54 bytes) 000810 1762490245.140794 trunc from 0x03005e to 0x00e991 (0x216cc bytes) 000811 1762490245.163808 write 0x017cf9 thru 0x01949b (0x17a3 bytes) 000812 1762490245.164809 mapread 0x003007 thru 0x00493b (0x1935 bytes) 000817 1762490245.169386 mapwrite 0x03dcb4 thru 0x03ffff (0x234c bytes) 000819 1762490245.334483 mapwrite 0x002ade thru 0x008827 (0x5d4a bytes) 000821 1762490245.432974 mapwrite 0x035bed thru 0x038a72 (0x2e86 bytes) 000823 1762490245.577163 trunc from 0x040000 to 0x00b5df (0x34a20 bytes) 000826 1762490245.642474 read 0x002c51 thru 0x00b5df (0x898f bytes) 000827 1762490245.646484 trunc from 0x00b5e0 to 0x033135 (0x27b56 bytes) 000831 1762490245.685033 mapread 0x016aae thru 0x023246 (0xc799 bytes) 000832 1762490245.732606 mapread 0x01704c thru 0x020124 (0x90d9 bytes) 000833 1762490245.740572 mapwrite 0x024c1d thru 0x030641 (0xba25 bytes) 000834 1762490245.834139 mapwrite 0x0017a0 thru 0x00ed7d (0xd5de bytes) 000835 1762490245.894956 mapread 0x025b28 thru 0x02ac2f (0x5108 bytes) 000836 1762490245.895723 mapread 0x02d22d thru 0x033135 (0x5f09 bytes) 000837 1762490245.896442 read 0x00a4a2 thru 0x00d252 (0x2db1 bytes) 000840 1762490245.900866 write 0x02f968 thru 0x0366a1 (0x6d3a bytes) 000841 1762490245.903366 read 0x0039e5 thru 0x00823a (0x4856 bytes) 000843 1762490245.910907 trunc from 0x0366a2 to 0x000c37 (0x35a6a bytes) 000844 1762490245.947740 read 0x000165 thru 0x000c37 (0xad3 bytes) 000847 1762490245.949322 trunc from 0x000c38 to 0x003b08 (0x2ed1 bytes) 000848 1762490245.976177 write 0x00cb48 thru 0x01123a (0x46f3 bytes) 000853 1762490245.985906 mapwrite 0x02919f thru 0x030a13 (0x7875 bytes) 000854 1762490246.083424 mapwrite 0x03b360 thru 0x03ffff (0x4ca0 bytes) 000856 1762490246.185195 mapread 0x03dd56 thru 0x03ffff (0x22aa bytes) 000859 1762490246.193559 MIRROR_EXTEND to 2 mirrors 000862 1762490246.727197 read 0x01fab1 thru 0x02a83a (0xad8a bytes) 000863 1762490246.740007 read 0x039324 thru 0x03e2f8 (0x4fd5 bytes) 000864 1762490246.755029 mapwrite 0x03668a thru 0x03e048 (0x79bf bytes) 000865 1762490246.998335 mapwrite 0x03e152 thru 0x03ffff (0x1eae bytes) 000866 1762490247.072432 mapread 0x03be20 thru 0x03ffff (0x41e0 bytes) 000867 1762490247.073142 mapread 0x038b91 thru 0x03ffff (0x746f bytes) 000868 1762490247.077903 MIRROR_RESYNC 2 mirrors 000870 1762490247.778561 write 0x00e8b7 thru 0x016f46 (0x8690 bytes) 000871 1762490247.866714 MIRROR_EXTEND to 3 mirrors 000872 1762490248.489840 read 0x0009a3 thru 0x00f769 (0xedc7 bytes) 000875 1762490248.561490 MIRROR_EXTEND to 4 mirrors 000876 1762490249.150789 mapwrite 0x00f9f4 thru 0x017098 (0x76a5 bytes) 000877 1762490249.314035 trunc from 0x040000 to 0x03395b (0xc6a4 bytes) 000878 1762490249.346072 read 0x019775 thru 0x028d06 (0xf592 bytes) 000881 1762490249.363987 MIRROR_RESYNC 4 mirrors 000882 1762490250.130886 read 0x02a560 thru 0x03395b (0x93fc bytes) 000883 1762490250.141635 mapwrite 0x020062 thru 0x021308 (0x12a7 bytes) 000884 1762490250.276734 mapread 0x021763 thru 0x023ef1 (0x278f bytes) 000886 1762490250.278135 trunc from 0x03395c to 0x02fca0 (0x3cbb bytes) 000888 1762490250.326295 mapwrite 0x0118c9 thru 0x018758 (0x6e90 bytes) 000889 1762490250.382276 trunc from 0x02fca1 to 0x0249c9 (0xb2d7 bytes) 000890 1762490250.402369 mapwrite 0x033273 thru 0x03e80b (0xb599 bytes) 000892 1762490250.477980 mapwrite 0x02b566 thru 0x033f71 (0x8a0c bytes) 000893 1762490250.554517 write 0x0309cd thru 0x03bce3 (0xb317 bytes) 000894 1762490250.555889 mapread 0x01d633 thru 0x01ef22 (0x18f0 bytes) 000895 1762490250.561439 MIRROR_EXTEND to 5 mirrors 000896 1762490251.318239 mapread 0x026255 thru 0x02affa (0x4da6 bytes) 000897 1762490251.355400 trunc from 0x03e80c to 0x0017db (0x3d030 bytes) 000898 1762490251.432586 mapread 0x000363 thru 0x0017db (0x1479 bytes) 000900 1762490251.441807 read 0x001639 thru 0x001796 (0x15e bytes) 000905 1762490251.445545 read 0x00168c thru 0x0017db (0x150 bytes) 000906 1762490251.447566 read 0x001035 thru 0x0017db (0x7a7 bytes) 000907 1762490251.454067 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000910 1762490251.722822 mapwrite 0x00b2a8 thru 0x016b69 (0xb8c2 bytes) 000911 1762490251.847637 mapwrite 0x034d38 thru 0x035575 (0x83e bytes) 000913 1762490251.983023 MIRROR_RESYNC 4 mirrors 000914 1762490252.904859 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000915 1762490253.261749 mapwrite 0x00f7e2 thru 0x01f099 (0xf8b8 bytes) 000916 1762490253.499374 mapwrite 0x0255b4 thru 0x02a64f (0x509c bytes) 000918 1762490253.589780 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000920 1762490253.825534 trunc from 0x035576 to 0x02d1ba (0x83bb bytes) 000922 1762490253.858851 write 0x016648 thru 0x02538b (0xed44 bytes) 000923 1762490253.887184 trunc from 0x02d1bb to 0x0005a3 (0x2cc17 bytes) 000925 1762490253.908632 mapread 0x000409 thru 0x0005a3 (0x19b bytes) 000926 1762490253.919561 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000928 1762490254.057548 trunc from 0x0005a4 to 0x037619 (0x37076 bytes) 000929 1762490254.080166 mapread 0x021b85 thru 0x02d738 (0xbbb4 bytes) 000932 1762490254.112833 mapread 0x02f294 thru 0x037619 (0x8386 bytes) 000933 1762490254.118990 read 0x011ea6 thru 0x017b09 (0x5c64 bytes) 000935 1762490254.133733 MIRROR_EXTEND to 2 mirrors 000936 1762490254.831655 mapwrite 0x00ec45 thru 0x01b990 (0xcd4c bytes) 000937 1762490255.048036 mapwrite 0x00bc0e thru 0x00d0f3 (0x14e6 bytes) 000939 1762490255.120946 write 0x00ee5e thru 0x019f43 (0xb0e6 bytes) 000940 1762490255.134609 MIRROR_RESYNC 2 mirrors 000941 1762490255.909399 read 0x011a34 thru 0x019414 (0x79e1 bytes) 000942 1762490255.918480 mapwrite 0x003844 thru 0x011d54 (0xe511 bytes) 000943 1762490256.130937 write 0x0317df thru 0x033a7b (0x229d bytes) 000947 1762490256.147220 read 0x022501 thru 0x02b5a3 (0x90a3 bytes) 000949 1762490256.158032 trunc from 0x03761a to 0x033cfd (0x391c bytes) 000951 1762490256.207965 MIRROR_RESYNC 2 mirrors 000952 1762490256.912062 trunc from 0x033cfe to 0x0286cf (0xb62e bytes) 000953 1762490257.020057 trunc from 0x0286d0 to 0x037995 (0xf2c6 bytes) 000954 1762490257.039940 mapwrite 0x0074d4 thru 0x00e9df (0x750c bytes) 000955 1762490257.116169 read 0x0017d0 thru 0x004de4 (0x3615 bytes) 000957 1762490257.122183 mapread 0x035895 thru 0x037995 (0x2101 bytes) 000958 1762490257.132356 mapread 0x00a45e thru 0x018a05 (0xe5a8 bytes) 000960 1762490257.157186 mapwrite 0x03f052 thru 0x03ffff (0xfae bytes) 000961 1762490257.305581 MIRROR_RESYNC 2 mirrors 000962 1762490257.980387 trunc from 0x040000 to 0x03787b (0x8784 bytes) 000963 1762490258.089908 trunc from 0x03787c to 0x023de4 (0x13a97 bytes) 000965 1762490258.113685 trunc from 0x023de5 to 0x01ba82 (0x8362 bytes) 000967 1762490258.131433 read 0x002ebd thru 0x006713 (0x3857 bytes) 000969 1762490258.139462 trunc from 0x01ba83 to 0x00f2e4 (0xc79e bytes) 000970 1762490258.164399 MIRROR_RESYNC 2 mirrors 000972 1762490258.688735 write 0x01cd8c thru 0x021308 (0x457d bytes) 000973 1762490258.765349 read 0x01c090 thru 0x021308 (0x5279 bytes) 000974 1762490258.766124 read 0x0017e4 thru 0x00f3d5 (0xdbf2 bytes) 000975 1762490258.777293 write 0x0046aa thru 0x00a779 (0x60d0 bytes) 000976 1762490258.785350 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000977 1762490258.960439 mapread 0x00d455 thru 0x01c5b5 (0xf161 bytes) 000980 1762490258.991330 write 0x02a857 thru 0x0330a7 (0x8851 bytes) 000981 1762490258.993062 mapread 0x02e7ac thru 0x0330a7 (0x48fc bytes) 000982 1762490258.994719 mapwrite 0x02f356 thru 0x02fae7 (0x792 bytes) 000983 1762490259.047123 read 0x00bcbb thru 0x0187db (0xcb21 bytes) 000984 1762490259.049019 mapwrite 0x036118 thru 0x03ba5c (0x5945 bytes) 000985 1762490259.156381 mapwrite 0x01bcbb thru 0x028597 (0xc8dd bytes) 000987 1762490259.259698 write 0x021a18 thru 0x028d1e (0x7307 bytes) 000988 1762490259.261622 trunc from 0x03ba5d to 0x028aaf (0x12fad bytes) 000990 1762490259.287683 read 0x0175f2 thru 0x01baaf (0x44be bytes) 000991 1762490259.288650 read 0x00ef4a thru 0x01afdc (0xc093 bytes) 000992 1762490259.289178 mapread 0x00b154 thru 0x00bdd7 (0xc84 bytes) 000994 1762490259.290817 mapwrite 0x01431a thru 0x01a008 (0x5cef bytes) 000997 1762490259.343123 mapwrite 0x02a10d thru 0x030550 (0x6444 bytes) 000998 1762490259.430853 MIRROR_EXTEND to 2 mirrors All operations completed A-OK! PASS 70a (76s) == sanity-flr test 71: check mirror extend parent fid ==== 23:37:45 (1762490265) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 200a: stress test ===================== 23:37:48 (1762490268) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1952992 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3395616 .. failed Extending file size to 2475424 .. Extending file size to 5028704 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5409440 .. failed Extending file size to 7945024 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8534944 .. Extending file size to 8272064 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8325664 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8857888 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8625184 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9117024 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 54837 54838 54840 54841 54842 failed 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:39:05 (1762490345) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8315904 .. Extending file size to 6543328 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7926400 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8730720 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8840704 .. Extending file size to 8081696 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8763296 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9121952 .. Extending file size to 9189280 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 56738 56740 56741 56742 56743 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:40:24 (1762490424) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11549 s, 18.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.475206 s, 4.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:40:45 (1762490445) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164476 s, 12.8 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 23:40:55 (1762490455) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ea:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x285:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x27f:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:41:05 (1762490465) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.277857 s, 7.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x286:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x281:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:41:13 (1762490473) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.260205 s, 8.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x288:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x282:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x287:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x288:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x282:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x283:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:41:23 (1762490483) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.324918 s, 6.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x284:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x284:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x285:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:41:58 (1762490518) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210557 s, 5.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x286:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x286:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } PASS 204d (61s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:42:59 (1762490579) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x289:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.345966 s, 6.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x289:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x289:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:44:03 (1762490643) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. Creating new pool test_204f oleg413-server: Pool lustre.test_204f created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.238411 s, 8.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg413-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:45:22 (1762490722) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:45:32 (1762490732) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:45:43 (1762490743) oleg413-server: Pool lustre.test_206 created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg413-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:46:11 (1762490771) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749534 s, 14.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485768 s, 21.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7351d946 Mirror 3: 0x7351d946 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:46:23 (1762490783) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:46:27 (1762490787) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:46:30 (1762490790) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.136267 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1480000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x16a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x16a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x16a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1240000 Read this (4 bytes): BBBB 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:47:11 (1762490831) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0202217 s, 405 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:47:19 (1762490839) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315536 s, 33.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20f lmm_fid: [0x200000404:0x20f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x20f lmm_fid: [0x200000404:0x20f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x297:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834447 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-11-06 23:47:21.000000000 -0500 Modify: 2025-11-06 23:47:21.000000000 -0500 Change: 2025-11-06 23:47:22.000000000 -0500 Birth: 2025-11-06 23:47:21.000000000 -0500 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:47:28 (1762490848) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:47:32 (1762490852) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.549814 s, 74.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0870939 s, 47.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Nov 6 23:47 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x211:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Nov 6 23:47 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test complete, duration 1610 sec =========== 23:47:41 (1762490861) === sanity-flr: start cleanup 23:47:42 (1762490862) === === sanity-flr: finish cleanup 23:47:48 (1762490868) ===