-----============= acceptance-small: sanity-flr ============----- Thu Sep 25 09:45:35 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg244-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:45:42 (1758807942) === oleg244-client.virtnet: executing check_config_client /mnt/lustre oleg244-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg244-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b8391c3b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b8391c3b800.idle_timeout=debug disable quota as required oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:45:55 (1758807955) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7538] == sanity-flr test 0a: lfs mirror create with -N option == 09:45:56 (1758807956) PASS 0a (6s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:46:02 (1758807962) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:46:03 (1758807963) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:46:05 (1758807965) PASS 0d (8s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:46:13 (1758807973) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:46:14 (1758807974) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 09:46:16 (1758807976) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b8391c3b800.stats=clear osc.lustre-OST0001-osc-ffff9b8391c3b800.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:46:19 (1758807979) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:46:23 (1758807983) PASS 0j (3s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:46:26 (1758807986) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:46:28 (1758807988) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 09:46:31 (1758807991) PASS 2 (3s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:46:34 (1758807994) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:46:36 (1758807996) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:46:39 (1758807999) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:46:43 (1758808003) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:46:45 (1758808005) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.551344 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784628 s, 13.4 MB/s PASS 21 (5s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:46:50 (1758808010) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0256959 s, 81.6 MB/s PASS 22 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:46:53 (1758808013) Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b8391c3b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b8391c3b800.ost_server_uuid in DISCONN state after 3 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b8391c3b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b8391c3b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg244-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (32s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:47:25 (1758808045) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.20964 s, 50.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b8391c3b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b8391c3b800.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:47:41 (1758808061) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b8391c3b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b8391c3b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b8391c3b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b8391c3b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg244-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (28s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:48:09 (1758808089) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.07278 s, 48.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 25 09:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.340s user 0m0.001s sys 0m0.161s Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b8391c3b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b8391c3b800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... real 0m0.406s user 0m0.001s sys 0m0.156s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b8391c3b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b8391c3b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.416s user 0m0.000s sys 0m0.167s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg244-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg244-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b8391c3b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b8391c3b800.ost_server_uuid in FULL state after 0 sec PASS 33b (34s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:48:44 (1758808124) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:48:45 (1758808125) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:48:47 (1758808127) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 09:48:48 (1758808128) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 09:48:52 (1758808132) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.272273 s, 46.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:48:58 (1758808138) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019917 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147971 s, 70.9 MB/s PASS 36b (4s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:49:02 (1758808142) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:49:06 (1758808146) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00264618 s, 387 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000633936 s, 1.6 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000780437 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000805462 s, 1.3 MB/s flv=10 olv=10 PASS 36d (5s) == sanity-flr test 37: mirror I/O API verification ======= 09:49:11 (1758808151) create test files with size 15733522 .. 16860fa322a11de9a2defa6e729b4bf6 - d7df00d78521da1c5f72c97facd7aef2 - 3f9dc4bd0b8d16733f067cbe8a3ce338 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg244-client.virtnet /mnt/lustre (opts:) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre PASS 37 (10s) == sanity-flr test 38: resync ============================ 09:49:21 (1758808161) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:49:27 (1758808167) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:49:30 (1758808170) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644889 s, 16.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137245 s, 76.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 09:49:34 (1758808174) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0574457 s, 36.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.154374 s, 27.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0847224 s, 24.8 MB/s **only resync mirror 2 **resync mirror 2 again **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 09:49:39 (1758808179) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 09:49:40 (1758808180) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:49:41 (1758808181) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139653 s, 75.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 09:49:45 (1758808185) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:49:46 (1758808186) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:49:50 (1758808190) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.118109 s, 88.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg244-client.virtnet /mnt/lustre (opts:) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:50:09 (1758808209) llite.lustre-ffff9b8391c3b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0965185 s, 109 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg244-client.virtnet /mnt/lustre (opts:) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b83902bf800.xattr_cache=1 PASS 44e (5s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:50:14 (1758808214) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (14s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:50:28 (1758808228) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (14s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:50:42 (1758808242) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 09:50:43 (1758808243) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0835204 s, 37.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0752284 s, 41.8 MB/s ** resync it again ** verify mirror 2 doesn't change original checksum: 3dd36fbaacfa005565e2605dc941b3ba - mirror 1 checksum: f58d08c698a4a6de1a29ae3f11e1d86d - mirror 2 checksum: 3dd36fbaacfa005565e2605dc941b3ba - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Sep 25 09:50:44 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Sep 25 09:50:44 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3dd36fbaacfa005565e2605dc941b3ba - mirror 1 checksum: f58d08c698a4a6de1a29ae3f11e1d86d - mirror 2 checksum: f58d08c698a4a6de1a29ae3f11e1d86d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Sep 25 09:50:44 2025' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Sep 25 09:50:44 2025' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:50:48 (1758808248) Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 09:50:50 (1758808250) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 09:51:05 (1758808265) targets are mounted 09:51:05 (1758808265) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171231 s, 61.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (24s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:51:12 (1758808272) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:51:13 (1758808273) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:51:15 (1758808275) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00409888 s, 4.0 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:51:17 (1758808277) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:51:19 (1758808279) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:51:20 (1758808280) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:51:29 (1758808289) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:51:50 (1758808310) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:52:08 (1758808328) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (19s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:52:27 (1758808347) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (3s) == sanity-flr test 70: mirror create and split race ====== 09:52:30 (1758808350) mirror create pid 52201 mirror split pid 52202 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 52201 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4975: 52202 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 09:53:33 (1758808413) Seed set to 8414 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1758808414.364303 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0xdaa3 000003 1758808414.548138 trunc from 00000000 to 0x00daa2 (0xdaa3 bytes) 000004 1758808414.591814 mapwrite 0x028290 thru 0x034460 (0xc1d1 bytes) 000005 1758808414.657320 write 0x025703 thru 0x030929 (0xb227 bytes) 000008 1758808414.661049 MIRROR_RESYNC 2 mirrors 000010 1758808414.909914 MIRROR_RESYNC 2 mirrors 000012 1758808415.024717 read 0x026414 thru 0x034460 (0xe04d bytes) 000013 1758808415.030548 mapwrite 0x021878 thru 0x0237ea (0x1f73 bytes) truncating to largest ever: 0x21550 000016 1758808415.127971 trunc from 0x034461 to 0x02154f (0x12f11 bytes) 000018 1758808415.159340 write 0x030761 thru 0x03e072 (0xd912 bytes) 000019 1758808415.161166 mapwrite 0x01d666 thru 0x029a8b (0xc426 bytes) 000022 1758808415.213900 trunc from 0x03e073 to 0x017e8a (0x261e8 bytes) 000024 1758808415.229842 mapread 0x00b15c thru 0x00c5b4 (0x1459 bytes) 000025 1758808415.239471 read 0x007dd3 thru 0x00c5a8 (0x47d6 bytes) 000026 1758808415.240074 mapread 0x007f7d thru 0x0111b9 (0x923d bytes) 000027 1758808415.240868 mapwrite 0x0221ef thru 0x03156a (0xf37c bytes) 000029 1758808415.304115 mapread 0x02736d thru 0x03156a (0xa1fe bytes) 000032 1758808415.310741 MIRROR_EXTEND to 3 mirrors 000034 1758808415.586611 trunc from 0x03156b to 0x00bf83 (0x255e7 bytes) 000035 1758808415.634641 write 0x001f08 thru 0x00250c (0x605 bytes) 000036 1758808415.641761 mapwrite 0x016e4f thru 0x01d08e (0x6240 bytes) 000037 1758808415.695327 mapread 0x003dd2 thru 0x006a92 (0x2cc1 bytes) 000039 1758808415.701087 mapwrite 0x0104cd thru 0x01946a (0x8f9e bytes) 000040 1758808415.729959 trunc from 0x01d08f to 0x014380 (0x8d0e bytes) truncating to largest ever: 0x23590 000043 1758808415.740716 trunc from 0x014381 to 0x02358f (0xf20f bytes) 000044 1758808415.752889 read 0x00efc4 thru 0x01940e (0xa44b bytes) 000048 1758808415.759391 write 0x013439 thru 0x01ff9b (0xcb63 bytes) 000049 1758808415.765088 read 0x00ba59 thru 0x01393c (0x7ee4 bytes) 000050 1758808415.765807 write 0x03ae6a thru 0x03fb79 (0x4d10 bytes) 000051 1758808415.766901 trunc from 0x03fb7a to 0x004ff2 (0x3ab87 bytes) 000052 1758808415.778778 write 0x00685f thru 0x00e221 (0x79c3 bytes) 000053 1758808415.779990 trunc from 0x00e222 to 0x00f7fa (0x15d9 bytes) 000054 1758808415.790056 read 0x008873 thru 0x00f7fa (0x6f88 bytes) 000056 1758808415.795812 mapwrite 0x039fb2 thru 0x03d0d9 (0x3128 bytes) 000057 1758808415.851031 write 0x0115be thru 0x01d723 (0xc166 bytes) 000058 1758808415.856185 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x2d197 000059 1758808416.184322 trunc from 0x03d0da to 0x02d196 (0xff43 bytes) 000060 1758808416.221306 write 0x02900a thru 0x029460 (0x457 bytes) 000061 1758808416.225058 mapread 0x022b48 thru 0x02d196 (0xa64f bytes) 000062 1758808416.237789 mapwrite 0x01c1cc thru 0x021573 (0x53a8 bytes) 000064 1758808416.265032 trunc from 0x02d197 to 0x00d008 (0x2018e bytes) 000067 1758808416.284262 MIRROR_EXTEND to 4 mirrors 000068 1758808416.586828 MIRROR_RESYNC 4 mirrors 000069 1758808416.868600 mapwrite 0x030c75 thru 0x03d48a (0xc816 bytes) 000071 1758808416.966014 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000072 1758808417.073335 mapread 0x0296f0 thru 0x0346d6 (0xafe7 bytes) 000075 1758808417.089088 mapread 0x0289ac thru 0x030a9f (0x80f4 bytes) 000077 1758808417.090338 read 0x022324 thru 0x02b95f (0x963c bytes) 000079 1758808417.097651 MIRROR_RESYNC 3 mirrors 000080 1758808417.410464 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000081 1758808417.532954 mapread 0x029525 thru 0x037f72 (0xea4e bytes) 000084 1758808417.546297 mapread 0x013daf thru 0x01bad2 (0x7d24 bytes) 000085 1758808417.551554 read 0x007c2e thru 0x00fa62 (0x7e35 bytes) 000086 1758808417.555748 mapwrite 0x00c50f thru 0x0150eb (0x8bdd bytes) truncating to largest ever: 0x3b095 000087 1758808417.644178 trunc from 0x03d48b to 0x03b094 (0x23f6 bytes) 000088 1758808417.656011 read 0x02d6d8 thru 0x0392f0 (0xbc19 bytes) 000089 1758808417.662833 mapread 0x020705 thru 0x022878 (0x2174 bytes) 000091 1758808417.666516 MIRROR_RESYNC 2 mirrors 000093 1758808417.853622 mapread 0x02f783 thru 0x03b094 (0xb912 bytes) 000095 1758808417.864061 mapwrite 0x003aa3 thru 0x00f948 (0xbea6 bytes) 000096 1758808417.934655 mapwrite 0x0298bb thru 0x036168 (0xc8ae bytes) 000097 1758808417.968497 read 0x01a991 thru 0x0210f2 (0x6762 bytes) 000098 1758808417.973037 trunc from 0x03b095 to 0x02bc32 (0xf462 bytes) 000099 1758808417.992521 trunc from 0x02bc33 to 0x0123b4 (0x1987e bytes) 000101 1758808417.999386 mapread 0x006a3e thru 0x00870f (0x1cd2 bytes) 000102 1758808418.000197 trunc from 0x0123b5 to 0x036738 (0x24384 bytes) 000103 1758808418.008188 trunc from 0x036739 to 0x017f92 (0x1e7a6 bytes) 000104 1758808418.015212 mapwrite 0x03dfb0 thru 0x03f2a1 (0x12f2 bytes) 000105 1758808418.043618 trunc from 0x03f2a2 to 0x010e7a (0x2e427 bytes) 000106 1758808418.052749 mapread 0x00e947 thru 0x010e7a (0x2534 bytes) 000107 1758808418.053688 trunc from 0x010e7b to 0x00aaba (0x63c0 bytes) 000108 1758808418.062119 read 0x00793e thru 0x00aaba (0x317d bytes) 000109 1758808418.065035 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000110 1758808418.134024 trunc from 0x00aabb to 0x02dd44 (0x2328a bytes) 000111 1758808418.142898 mapwrite 0x0255ba thru 0x026929 (0x1370 bytes) 000112 1758808418.170244 write 0x013908 thru 0x01758c (0x3c85 bytes) 000113 1758808418.177000 trunc from 0x02dd45 to 0x01031a (0x1da2a bytes) 000115 1758808418.186716 trunc from 0x01031b to 0x01a466 (0xa14c bytes) 000116 1758808418.194248 write 0x01234a thru 0x0135c4 (0x127b bytes) 000118 1758808418.203128 MIRROR_EXTEND to 2 mirrors 000119 1758808418.512759 MIRROR_EXTEND to 3 mirrors 000120 1758808418.787490 mapread 0x008b8c thru 0x0112b4 (0x8729 bytes) 000121 1758808418.801773 mapwrite 0x03090b thru 0x03372c (0x2e22 bytes) 000122 1758808418.854043 trunc from 0x03372d to 0x01460d (0x1f11f bytes) 000125 1758808418.865993 mapread 0x005ee5 thru 0x01460d (0xe729 bytes) 000126 1758808418.874938 mapwrite 0x015933 thru 0x022400 (0xcace bytes) 000127 1758808418.915373 read 0x006a58 thru 0x0078d4 (0xe7d bytes) 000128 1758808418.916117 read 0x008c01 thru 0x009567 (0x967 bytes) 000129 1758808418.916687 mapwrite 0x016a47 thru 0x01d154 (0x670e bytes) 000130 1758808418.943517 mapread 0x008a1b thru 0x00efb7 (0x659d bytes) 000132 1758808418.944746 trunc from 0x022401 to 0x002267 (0x20199 bytes) 000133 1758808418.957037 mapwrite 0x00c15a thru 0x01042c (0x42d3 bytes) 000134 1758808418.997238 trunc from 0x01042d to 0x0171ab (0x6d7f bytes) 000136 1758808419.007688 read 0x012513 thru 0x013a35 (0x1523 bytes) 000138 1758808419.014167 mapread 0x007e34 thru 0x013f44 (0xc111 bytes) 000139 1758808419.015083 trunc from 0x0171ac to 0x000487 (0x16d24 bytes) 000140 1758808419.028007 mapread 0x00029d thru 0x000487 (0x1eb bytes) 000141 1758808419.031490 MIRROR_RESYNC 3 mirrors 000142 1758808419.333797 read 0x000080 thru 0x000487 (0x408 bytes) 000143 1758808419.336542 write 0x02277d thru 0x02f07c (0xc900 bytes) 000145 1758808419.366381 trunc from 0x02f07d to 0x03287b (0x37ff bytes) 000146 1758808419.386167 read 0x01e040 thru 0x020a8b (0x2a4c bytes) 000147 1758808419.392523 read 0x01be8e thru 0x0203ad (0x4520 bytes) 000148 1758808419.393089 mapread 0x0242ef thru 0x02aa2a (0x673c bytes) 000151 1758808419.394716 mapwrite 0x00e2cb thru 0x0101cf (0x1f05 bytes) 000152 1758808419.426303 MIRROR_EXTEND to 4 mirrors 000153 1758808419.751372 trunc from 0x03287c to 0x001126 (0x31755 bytes) 000155 1758808419.820843 read 0x0004cc thru 0x001126 (0xc5b bytes) 000156 1758808419.824285 write 0x03dfd1 thru 0x03ffff (0x202f bytes) 000157 1758808419.825388 write 0x0240e3 thru 0x02f976 (0xb894 bytes) 000158 1758808419.837759 MIRROR_EXTEND to 5 mirrors 000161 1758808420.147644 write 0x011a5f thru 0x014e19 (0x33bb bytes) 000163 1758808420.193733 trunc from 0x040000 to 0x000f2a (0x3f0d5 bytes) 000164 1758808420.211518 mapread 0x0003fe thru 0x000f2a (0xb2d bytes) 000165 1758808420.223069 mapwrite 0x00c08d thru 0x018234 (0xc1a8 bytes) 000166 1758808420.275827 mapwrite 0x0318a4 thru 0x03d999 (0xc0f6 bytes) 000167 1758808420.320506 MIRROR_RESYNC 5 mirrors 000168 1758808420.724950 mapread 0x038764 thru 0x03d999 (0x5236 bytes) 000169 1758808420.734601 MIRROR_EXTEND to 6 mirrors 000170 1758808420.996751 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) truncating to largest ever: 0x3c5d9 000172 1758808421.166793 trunc from 0x03d99a to 0x03c5d8 (0x13c1 bytes) 000174 1758808421.235052 write 0x0091d2 thru 0x00cff1 (0x3e20 bytes) 000175 1758808421.242072 mapread 0x019282 thru 0x024918 (0xb697 bytes) 000177 1758808421.259727 MIRROR_EXTEND to 6 mirrors 000178 1758808421.526121 mapread 0x0062a4 thru 0x0145a5 (0xe302 bytes) 000180 1758808421.554086 mapwrite 0x029f65 thru 0x02c193 (0x222f bytes) 000181 1758808421.662599 read 0x01c5c2 thru 0x021f3c (0x597b bytes) 000182 1758808421.667971 write 0x036e68 thru 0x037f17 (0x10b0 bytes) 000183 1758808421.675518 MIRROR_RESYNC 6 mirrors 000185 1758808422.236866 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000186 1758808422.446908 trunc from 0x03c5d9 to 0x02daa3 (0xeb35 bytes) 000187 1758808422.500858 mapwrite 0x03fe65 thru 0x03ffff (0x19b bytes) 000188 1758808422.563236 read 0x021135 thru 0x024a56 (0x3922 bytes) 000189 1758808422.567362 read 0x02874d thru 0x0317ab (0x905f bytes) 000191 1758808422.576834 read 0x03bddc thru 0x03ffff (0x4224 bytes) 000193 1758808422.577870 read 0x0268a5 thru 0x032dcd (0xc529 bytes) 000194 1758808422.578376 read 0x02092f thru 0x0283ba (0x7a8c bytes) 000199 1758808422.580653 read 0x01dcda thru 0x029005 (0xb32c bytes) 000200 1758808422.581168 trunc from 0x040000 to 0x03a6ca (0x5935 bytes) 000201 1758808422.599937 trunc from 0x03a6cb to 0x00251f (0x381ab bytes) 000202 1758808422.613511 trunc from 0x002520 to 0x006810 (0x42f1 bytes) 000205 1758808422.626995 mapwrite 0x007fea thru 0x01552e (0xd545 bytes) 000209 1758808422.678692 mapwrite 0x01ba95 thru 0x022ea2 (0x740e bytes) 000210 1758808422.741187 mapread 0x01be78 thru 0x022ea2 (0x702b bytes) 000213 1758808422.743051 mapread 0x00e4ae thru 0x00f48a (0xfdd bytes) 000214 1758808422.744772 trunc from 0x022ea3 to 0x00ee96 (0x1400c bytes) 000216 1758808422.758383 read 0x004289 thru 0x009355 (0x50cd bytes) 000217 1758808422.759003 trunc from 0x00ee97 to 0x025c89 (0x16df3 bytes) truncating to largest ever: 0x3cfaf 000219 1758808422.774349 trunc from 0x025c8a to 0x03cfae (0x17325 bytes) 000220 1758808422.798784 read 0x01c661 thru 0x025942 (0x92e2 bytes) 000221 1758808422.807462 read 0x007995 thru 0x0120a8 (0xa714 bytes) 000222 1758808422.808033 trunc from 0x03cfaf to 0x029155 (0x13e59 bytes) 000225 1758808422.824032 write 0x027c7e thru 0x029de5 (0x2168 bytes) 000227 1758808422.825378 mapread 0x0159d6 thru 0x01aad1 (0x50fc bytes) 000228 1758808422.826169 trunc from 0x029de6 to 0x0238bd (0x6528 bytes) 000229 1758808422.841199 write 0x0307cd thru 0x032f70 (0x27a4 bytes) 000233 1758808422.844950 mapread 0x01b5f4 thru 0x029b36 (0xe543 bytes) 000234 1758808422.851055 write 0x031258 thru 0x03bd40 (0xaae9 bytes) 000239 1758808422.854083 write 0x01d03e thru 0x02483e (0x7801 bytes) 000241 1758808422.855511 write 0x017e2e thru 0x023e95 (0xc068 bytes) 000242 1758808422.856760 mapread 0x016cd5 thru 0x021490 (0xa7bc bytes) 000244 1758808422.858124 trunc from 0x03bd41 to 0x02cb5c (0xf1e4 bytes) 000246 1758808422.871623 trunc from 0x02cb5d to 0x037b3a (0xafde bytes) 000249 1758808422.883892 write 0x00638c thru 0x00b646 (0x52bb bytes) 000250 1758808422.884927 write 0x00a641 thru 0x00f843 (0x5203 bytes) 000251 1758808422.891637 MIRROR_RESYNC 5 mirrors 000252 1758808423.343376 read 0x02196e thru 0x0241af (0x2842 bytes) 000253 1758808423.347050 mapwrite 0x0208de thru 0x021947 (0x106a bytes) 000254 1758808423.434199 MIRROR_EXTEND to 6 mirrors 000255 1758808423.802817 mapread 0x01e870 thru 0x02df3f (0xf6d0 bytes) 000256 1758808423.829838 MIRROR_EXTEND to 7 mirrors 000257 1758808424.156047 read 0x01f848 thru 0x02e464 (0xec1d bytes) 000259 1758808424.164540 MIRROR_RESYNC 7 mirrors 000260 1758808424.722915 write 0x03c549 thru 0x03ffff (0x3ab7 bytes) truncating to largest ever: 0x3f9b9 000262 1758808424.757763 trunc from 0x040000 to 0x03f9b8 (0x647 bytes) 000263 1758808424.770846 trunc from 0x03f9b9 to 0x0353cc (0xa5ec bytes) 000264 1758808424.780009 mapwrite 0x004c2c thru 0x0088e9 (0x3cbe bytes) 000265 1758808424.826385 mapread 0x00a1ea thru 0x00ba6c (0x1883 bytes) 000267 1758808424.827691 mapwrite 0x01363d thru 0x016ecb (0x388f bytes) 000269 1758808424.866490 mapwrite 0x03737e thru 0x03ffff (0x8c82 bytes) 000270 1758808424.910548 trunc from 0x040000 to 0x035b91 (0xa46e bytes) 000272 1758808424.918615 mapread 0x02c384 thru 0x035b91 (0x980e bytes) 000273 1758808424.924398 trunc from 0x035b92 to 0x01410b (0x21a86 bytes) 000276 1758808424.936644 write 0x007e0b thru 0x00d31c (0x5512 bytes) 000277 1758808424.937845 write 0x0043c0 thru 0x00e10a (0x9d4b bytes) 000278 1758808424.938926 write 0x0085e6 thru 0x00b757 (0x3172 bytes) 000279 1758808424.939701 write 0x02a025 thru 0x03658c (0xc568 bytes) 000280 1758808424.940947 read 0x02912c thru 0x03658c (0xd461 bytes) 000281 1758808424.944306 trunc from 0x03658d to 0x03f257 (0x8ccb bytes) 000282 1758808424.952255 write 0x013bf4 thru 0x01de53 (0xa260 bytes) 000285 1758808424.957005 trunc from 0x03f258 to 0x010985 (0x2e8d2 bytes) 000286 1758808424.966067 read 0x00f620 thru 0x010985 (0x1366 bytes) 000287 1758808424.966696 read 0x002a8e thru 0x00b672 (0x8be5 bytes) 000288 1758808424.967184 write 0x01a84a thru 0x02106c (0x6823 bytes) 000290 1758808424.968358 read 0x01a604 thru 0x01ad5f (0x75c bytes) 000291 1758808424.968732 trunc from 0x02106d to 0x0117a4 (0xf8c8 bytes) 000292 1758808424.976091 write 0x03890e thru 0x03ffff (0x76f2 bytes) 000294 1758808424.977668 read 0x00a8a0 thru 0x00e0c0 (0x3821 bytes) 000295 1758808424.980464 MIRROR_EXTEND to 8 mirrors 000296 1758808425.318412 mapwrite 0x00bb3e thru 0x0155b1 (0x9a74 bytes) 000297 1758808425.453512 mapread 0x01dc67 thru 0x022c72 (0x500c bytes) 000298 1758808425.459625 write 0x02c5a6 thru 0x02de8c (0x18e7 bytes) 000300 1758808425.468719 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000302 1758808425.611134 trunc from 0x040000 to 0x02044a (0x1fbb5 bytes) 000303 1758808425.621087 mapread 0x01129f thru 0x0120bf (0xe21 bytes) 000304 1758808425.628006 read 0x00bcd7 thru 0x01bba2 (0xfecc bytes) 000315 1758808425.639032 mapread 0x00fcb0 thru 0x01c36f (0xc6c0 bytes) 000320 1758808425.647367 write 0x039019 thru 0x03c4c8 (0x34b0 bytes) 000321 1758808425.648457 mapread 0x036873 thru 0x03c4c8 (0x5c56 bytes) 000322 1758808425.655275 trunc from 0x03c4c9 to 0x02c989 (0xfb3f bytes) 000324 1758808425.665818 read 0x026ad4 thru 0x02c989 (0x5eb6 bytes) 000325 1758808425.674078 mapwrite 0x0017e1 thru 0x003871 (0x2091 bytes) 000326 1758808425.709857 MIRROR_RESYNC 7 mirrors 000327 1758808426.387068 mapwrite 0x017a97 thru 0x018a19 (0xf83 bytes) 000329 1758808426.486002 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000333 1758808426.646658 write 0x01c4c8 thru 0x02a038 (0xdb71 bytes) 000334 1758808426.655106 trunc from 0x02c98a to 0x0101db (0x1c7ae bytes) 000335 1758808426.669840 mapwrite 0x01dcfc thru 0x0237e3 (0x5ae8 bytes) 000336 1758808426.753035 read 0x0065ab thru 0x006b86 (0x5dc bytes) 000338 1758808426.758877 mapwrite 0x018dc4 thru 0x01d551 (0x478e bytes) 000339 1758808426.791541 mapread 0x013ad1 thru 0x02241d (0xe94d bytes) 000340 1758808426.803563 write 0x028be9 thru 0x034b50 (0xbf68 bytes) 000341 1758808426.804900 write 0x002b8b thru 0x011207 (0xe67d bytes) 000344 1758808426.820199 write 0x0307c4 thru 0x03e4af (0xdcec bytes) 000345 1758808426.822823 mapread 0x02857d thru 0x02e0aa (0x5b2e bytes) 000347 1758808426.824731 trunc from 0x03e4b0 to 0x0299ad (0x14b02 bytes) 000351 1758808426.838245 write 0x034584 thru 0x034fe2 (0xa5f bytes) 000352 1758808426.839951 write 0x0170b9 thru 0x01ecac (0x7bf4 bytes) 000353 1758808426.841085 mapwrite 0x004af2 thru 0x00e766 (0x9c75 bytes) 000356 1758808426.885717 mapwrite 0x02b24c thru 0x036221 (0xafd6 bytes) 000357 1758808426.956628 write 0x001de9 thru 0x00af5e (0x9176 bytes) 000358 1758808426.957809 read 0x02f672 thru 0x036221 (0x6bb0 bytes) 000359 1758808426.958363 read 0x020906 thru 0x02398e (0x3089 bytes) 000360 1758808426.967321 MIRROR_EXTEND to 7 mirrors 000362 1758808427.426334 trunc from 0x036222 to 0x0331e0 (0x3041 bytes) 000363 1758808427.488914 MIRROR_RESYNC 7 mirrors 000364 1758808428.181530 mapwrite 0x01e1b4 thru 0x029fa3 (0xbdf0 bytes) 000365 1758808428.280138 read 0x010bb1 thru 0x01a86d (0x9cbd bytes) 000367 1758808428.288328 mapread 0x00032b thru 0x003119 (0x2def bytes) 000368 1758808428.294119 read 0x015c1c thru 0x01ebe0 (0x8fc5 bytes) 000369 1758808428.297650 MIRROR_EXTEND to 8 mirrors 000370 1758808428.566020 MIRROR_RESYNC 8 mirrors 000371 1758808429.089883 mapread 0x01a26b thru 0x025a41 (0xb7d7 bytes) 000372 1758808429.112569 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000376 1758808429.320048 mapwrite 0x01e112 thru 0x01e58e (0x47d bytes) 000377 1758808429.435111 mapwrite 0x039431 thru 0x0396eb (0x2bb bytes) 000380 1758808429.501533 mapread 0x010fa4 thru 0x014137 (0x3194 bytes) 000382 1758808429.509870 mapread 0x02a191 thru 0x030459 (0x62c9 bytes) 000383 1758808429.517397 trunc from 0x0396ec to 0x0067a1 (0x32f4a bytes) 000386 1758808429.530083 trunc from 0x0067a2 to 0x00f15d (0x89bc bytes) 000388 1758808429.548178 read 0x006815 thru 0x0070ba (0x8a6 bytes) 000389 1758808429.560231 MIRROR_RESYNC 7 mirrors 000390 1758808430.225606 MIRROR_RESYNC 7 mirrors 000391 1758808430.376871 trunc from 0x00f15e to 0x02b4a3 (0x1c346 bytes) 000392 1758808430.450981 read 0x00bad5 thru 0x0147f6 (0x8d22 bytes) 000393 1758808430.459778 mapread 0x022a08 thru 0x026225 (0x381e bytes) 000395 1758808430.479597 trunc from 0x02b4a4 to 0x02a191 (0x1312 bytes) 000396 1758808430.499074 write 0x00843c thru 0x00fa97 (0x765c bytes) 000398 1758808430.511324 trunc from 0x02a192 to 0x03e854 (0x146c3 bytes) 000400 1758808430.525939 read 0x00098e thru 0x007896 (0x6f09 bytes) 000403 1758808430.536527 read 0x01b7b4 thru 0x025f08 (0xa755 bytes) 000404 1758808430.537270 mapwrite 0x03e93d thru 0x03ffff (0x16c3 bytes) 000406 1758808430.609105 mapwrite 0x015fcf thru 0x022d01 (0xcd33 bytes) 000407 1758808430.662342 mapread 0x032749 thru 0x03c147 (0x99ff bytes) 000412 1758808430.668071 read 0x002f4e thru 0x00860e (0x56c1 bytes) 000413 1758808430.668750 mapread 0x0364b0 thru 0x037033 (0xb84 bytes) 000415 1758808430.671135 read 0x01f1be thru 0x025b44 (0x6987 bytes) 000417 1758808430.673161 read 0x01d89c thru 0x0222fb (0x4a60 bytes) 000418 1758808430.673764 write 0x029d21 thru 0x0386b0 (0xe990 bytes) 000419 1758808430.675114 mapwrite 0x02e9eb thru 0x034e50 (0x6466 bytes) 000420 1758808430.722892 trunc from 0x040000 to 0x02710f (0x18ef0 bytes) 000421 1758808430.748734 MIRROR_EXTEND to 8 mirrors 000422 1758808431.156092 trunc from 0x027110 to 0x00fa28 (0x176e7 bytes) 000423 1758808431.292585 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000424 1758808431.494024 write 0x02a6ab thru 0x02c6a2 (0x1ff8 bytes) 000425 1758808431.495622 mapread 0x00e075 thru 0x0183f7 (0xa383 bytes) 000426 1758808431.523384 read 0x025489 thru 0x026b51 (0x16c9 bytes) 000427 1758808431.529040 read 0x012c30 thru 0x01b02d (0x83fe bytes) 000428 1758808431.546050 MIRROR_EXTEND to 8 mirrors 000429 1758808432.089610 trunc from 0x02c6a3 to 0x031292 (0x4bf0 bytes) 000430 1758808432.195104 read 0x00a189 thru 0x012a61 (0x88d9 bytes) 000433 1758808432.204062 read 0x01a4de thru 0x01ee04 (0x4927 bytes) 000434 1758808432.218733 trunc from 0x031293 to 0x0213e1 (0xfeb1 bytes) 000435 1758808432.238734 read 0x003d9d thru 0x005ada (0x1d3e bytes) 000436 1758808432.239499 mapread 0x018613 thru 0x019bbb (0x15a9 bytes) 000437 1758808432.240972 mapread 0x017a93 thru 0x0213e1 (0x994f bytes) 000438 1758808432.250047 MIRROR_RESYNC 8 mirrors 000439 1758808433.183209 trunc from 0x0213e2 to 0x02d214 (0xbe33 bytes) 000440 1758808433.286574 read 0x01748d thru 0x0268e9 (0xf45d bytes) 000442 1758808433.299108 read 0x0001a7 thru 0x0099a3 (0x97fd bytes) 000448 1758808433.313053 trunc from 0x02d215 to 0x00f23a (0x1dfda bytes) 000451 1758808433.347831 MIRROR_EXTEND to 9 mirrors 000453 1758808433.811055 MIRROR_EXTEND to 10 mirrors 000456 1758808434.200330 trunc from 0x00f23b to 0x02ac29 (0x1b9ef bytes) 000457 1758808434.282275 MIRROR_EXTEND to 11 mirrors 000458 1758808434.704613 read 0x00a451 thru 0x00d767 (0x3317 bytes) 000460 1758808434.719184 write 0x02fe0d thru 0x03ded0 (0xe0c4 bytes) 000463 1758808434.793626 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000464 1758808434.961082 mapwrite 0x014230 thru 0x016165 (0x1f36 bytes) 000465 1758808435.009457 mapread 0x02b49e thru 0x0362e3 (0xae46 bytes) 000468 1758808435.029630 mapread 0x00ad72 thru 0x0111ba (0x6449 bytes) 000469 1758808435.035376 mapwrite 0x000d80 thru 0x001e39 (0x10ba bytes) 000470 1758808435.070096 read 0x02e9d9 thru 0x032652 (0x3c7a bytes) 000471 1758808435.071849 mapread 0x008424 thru 0x00f5f2 (0x71cf bytes) 000473 1758808435.073372 mapwrite 0x03d55b thru 0x03ffff (0x2aa5 bytes) 000474 1758808435.129374 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000475 1758808435.280812 mapwrite 0x00aed0 thru 0x0198e2 (0xea13 bytes) 000477 1758808435.370811 write 0x008fed thru 0x010c49 (0x7c5d bytes) 000479 1758808435.372410 mapwrite 0x01be7a thru 0x021543 (0x56ca bytes) 000480 1758808435.420731 read 0x03b2cb thru 0x03be1f (0xb55 bytes) 000481 1758808435.425342 mapread 0x016092 thru 0x025add (0xfa4c bytes) 000482 1758808435.426732 trunc from 0x040000 to 0x018bda (0x27425 bytes) 000486 1758808435.440917 mapwrite 0x01e224 thru 0x0268e8 (0x86c5 bytes) 000487 1758808435.503625 mapread 0x0183b6 thru 0x019f70 (0x1bbb bytes) 000488 1758808435.504577 mapwrite 0x01d63a thru 0x02af8b (0xd952 bytes) 000489 1758808435.565145 read 0x01a508 thru 0x01cb42 (0x263b bytes) 000491 1758808435.566331 read 0x00439c thru 0x005d34 (0x1999 bytes) 000492 1758808435.573196 read 0x01eee9 thru 0x027bb8 (0x8cd0 bytes) 000493 1758808435.573874 read 0x002d91 thru 0x00511d (0x238d bytes) 000494 1758808435.574414 mapread 0x0030db thru 0x00d7bd (0xa6e3 bytes) 000495 1758808435.575345 trunc from 0x02af8c to 0x0305cd (0x5642 bytes) 000497 1758808435.591592 MIRROR_EXTEND to 10 mirrors 000499 1758808435.977798 mapread 0x01788a thru 0x023656 (0xbdcd bytes) 000502 1758808436.030375 mapwrite 0x026ddb thru 0x02ef58 (0x817e bytes) 000503 1758808436.165747 mapread 0x025519 thru 0x02f560 (0xa048 bytes) 000505 1758808436.172269 write 0x02e05c thru 0x03a43b (0xc3e0 bytes) 000507 1758808436.174093 trunc from 0x03a43c to 0x008341 (0x320fa bytes) 000508 1758808436.208716 MIRROR_EXTEND to 11 mirrors 000509 1758808436.617478 MIRROR_RESYNC 11 mirrors 000510 1758808437.600493 mapwrite 0x0342c1 thru 0x039316 (0x5056 bytes) 000512 1758808437.751184 read 0x01a32f thru 0x0229dc (0x86ae bytes) 000514 1758808437.764822 read 0x027896 thru 0x03571f (0xde8a bytes) 000518 1758808437.787120 trunc from 0x039317 to 0x030536 (0x8de0 bytes) 000519 1758808437.817982 mapwrite 0x01d353 thru 0x025e3c (0x8aea bytes) 000522 1758808437.887783 write 0x0325e0 thru 0x032fb4 (0x9d5 bytes) 000523 1758808437.888752 read 0x02930a thru 0x02d48c (0x4183 bytes) 000525 1758808437.891277 trunc from 0x032fb5 to 0x00def1 (0x250c3 bytes) 000526 1758808437.914153 mapread 0x00bf8e thru 0x00def1 (0x1f64 bytes) 000528 1758808437.915603 mapread 0x00adac thru 0x00def1 (0x3146 bytes) 000530 1758808437.917346 trunc from 0x00def2 to 0x01e07a (0x10189 bytes) 000531 1758808437.936287 write 0x0255e1 thru 0x02db10 (0x8530 bytes) 000532 1758808437.937642 trunc from 0x02db11 to 0x026b5c (0x6fb4 bytes) 000534 1758808437.961590 read 0x00cfe0 thru 0x01727d (0xa29e bytes) 000535 1758808437.979900 read 0x0021af thru 0x005354 (0x31a6 bytes) 000536 1758808437.981384 mapread 0x0011f3 thru 0x0055de (0x43ec bytes) 000537 1758808437.982761 mapread 0x00304c thru 0x009c96 (0x6c4b bytes) 000538 1758808437.984774 trunc from 0x026b5d to 0x02d13e (0x65e2 bytes) 000539 1758808438.003629 MIRROR_RESYNC 11 mirrors 000541 1758808439.133812 read 0x000668 thru 0x00221d (0x1bb6 bytes) 000543 1758808439.140573 mapwrite 0x03e0a2 thru 0x03ffff (0x1f5e bytes) 000544 1758808439.291668 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000546 1758808439.482325 trunc from 0x040000 to 0x03c866 (0x3799 bytes) 000549 1758808439.498254 write 0x01a175 thru 0x01d869 (0x36f5 bytes) 000551 1758808439.515398 trunc from 0x03c867 to 0x014f43 (0x27923 bytes) 000552 1758808439.546097 mapread 0x00c729 thru 0x00fd22 (0x35fa bytes) 000553 1758808439.559559 read 0x00a79a thru 0x012fb8 (0x881f bytes) 000557 1758808439.565480 MIRROR_EXTEND to 11 mirrors 000560 1758808440.001555 read 0x001052 thru 0x001a3b (0x9ea bytes) 000564 1758808440.022462 read 0x008ddc thru 0x00d2dd (0x4502 bytes) 000566 1758808440.037225 trunc from 0x014f44 to 0x0052d5 (0xfc6e bytes) 000567 1758808440.141495 read 0x0026ce thru 0x0052d5 (0x2c08 bytes) 000569 1758808440.154962 read 0x0028a1 thru 0x0052d5 (0x2a35 bytes) 000570 1758808440.155598 write 0x023d28 thru 0x02b9c0 (0x7c99 bytes) 000572 1758808440.157532 read 0x00333d thru 0x00e0bc (0xad80 bytes) 000573 1758808440.173372 MIRROR_EXTEND to 12 mirrors 000577 1758808440.626384 mapread 0x00775b thru 0x013ce3 (0xc589 bytes) 000578 1758808440.668925 MIRROR_RESYNC 12 mirrors 000579 1758808441.742384 mapwrite 0x023f81 thru 0x0252e2 (0x1362 bytes) 000582 1758808441.920523 mapread 0x005f5e thru 0x01381a (0xd8bd bytes) 000583 1758808441.938386 read 0x005681 thru 0x006ec4 (0x1844 bytes) 000584 1758808441.939014 mapwrite 0x039d67 thru 0x03ffff (0x6299 bytes) 000587 1758808442.031251 write 0x01bbd3 thru 0x01dff2 (0x2420 bytes) 000588 1758808442.037959 trunc from 0x040000 to 0x038a84 (0x757b bytes) 000591 1758808442.061475 mapread 0x0131dd thru 0x01339d (0x1c1 bytes) 000593 1758808442.062849 mapread 0x025715 thru 0x032a0e (0xd2fa bytes) 000594 1758808442.070359 mapwrite 0x0277d0 thru 0x02b6d2 (0x3f03 bytes) 000596 1758808442.109670 mapread 0x0356cf thru 0x038a84 (0x33b6 bytes) 000597 1758808442.110702 mapread 0x000a70 thru 0x0010b5 (0x646 bytes) 000599 1758808442.118201 mapread 0x01c4af thru 0x01feff (0x3a51 bytes) 000600 1758808442.119136 mapread 0x02ec04 thru 0x0325ff (0x39fc bytes) 000602 1758808442.121226 mapread 0x006721 thru 0x007b4a (0x142a bytes) 000604 1758808442.123005 trunc from 0x038a85 to 0x003bad (0x34ed7 bytes) 000605 1758808442.141967 write 0x017280 thru 0x0200fe (0x8e7f bytes) 000606 1758808442.145126 write 0x021fcf thru 0x025c27 (0x3c59 bytes) 000607 1758808442.146381 mapwrite 0x002ccc thru 0x00b8e9 (0x8c1e bytes) 000608 1758808442.205132 MIRROR_RESYNC 12 mirrors 000609 1758808443.518049 mapread 0x0000a8 thru 0x00d80f (0xd768 bytes) 000610 1758808443.554110 read 0x00583e thru 0x00eff1 (0x97b4 bytes) 000611 1758808443.554720 read 0x0197da thru 0x021a72 (0x8299 bytes) 000612 1758808443.563386 mapwrite 0x02b073 thru 0x0399fa (0xe988 bytes) 000617 1758808443.723165 mapwrite 0x030efd thru 0x035f89 (0x508d bytes) 000619 1758808443.774359 read 0x0166cf thru 0x0227c3 (0xc0f5 bytes) 000620 1758808443.786516 trunc from 0x0399fb to 0x00041a (0x395e0 bytes) 000621 1758808443.810563 trunc from 0x00041b to 0x0375f3 (0x371d9 bytes) 000623 1758808443.833996 MIRROR_EXTEND to 13 mirrors 000624 1758808444.284024 mapread 0x01105e thru 0x01b47d (0xa420 bytes) 000626 1758808444.320681 write 0x0396e3 thru 0x03ffff (0x691d bytes) 000627 1758808444.375460 mapread 0x036f68 thru 0x03d64e (0x66e7 bytes) 000629 1758808444.382220 mapwrite 0x0008b5 thru 0x00ff91 (0xf6dd bytes) 000630 1758808444.428546 trunc from 0x040000 to 0x007523 (0x38adc bytes) 000632 1758808444.446949 read 0x005ccf thru 0x00700f (0x1341 bytes) 000636 1758808444.449064 trunc from 0x007524 to 0x00d565 (0x6042 bytes) 000638 1758808444.465233 trunc from 0x00d566 to 0x013103 (0x5b9e bytes) 000639 1758808444.479067 trunc from 0x013104 to 0x03e783 (0x2b680 bytes) 000641 1758808444.492469 trunc from 0x03e784 to 0x0353cf (0x93b4 bytes) 000642 1758808444.503579 mapwrite 0x01b667 thru 0x01d47c (0x1e16 bytes) 000645 1758808444.538719 trunc from 0x0353d0 to 0x00f4aa (0x25f25 bytes) 000646 1758808444.550864 write 0x01eadb thru 0x023c9e (0x51c4 bytes) 000647 1758808444.552032 trunc from 0x023c9f to 0x015947 (0xe357 bytes) 000648 1758808444.568883 read 0x00b32a thru 0x00c780 (0x1457 bytes) 000649 1758808444.576209 read 0x00702e thru 0x00b454 (0x4427 bytes) 000650 1758808444.576884 mapread 0x012b78 thru 0x015947 (0x2dd0 bytes) 000651 1758808444.577687 read 0x003095 thru 0x0055e9 (0x2555 bytes) 000652 1758808444.578217 read 0x00d286 thru 0x015947 (0x86c2 bytes) 000653 1758808444.578751 trunc from 0x015948 to 0x004171 (0x117d6 bytes) 000654 1758808444.600409 write 0x03f3bb thru 0x03fcbb (0x901 bytes) 000655 1758808444.605237 MIRROR_EXTEND to 14 mirrors 000657 1758808445.022152 trunc from 0x03fcbc to 0x029cf7 (0x15fc4 bytes) 000659 1758808445.106831 trunc from 0x029cf8 to 0x03748f (0xd798 bytes) 000661 1758808445.125039 mapread 0x0155d5 thru 0x01e052 (0x8a7e bytes) 000663 1758808445.144101 write 0x03025a thru 0x030fda (0xd81 bytes) 000664 1758808445.151185 trunc from 0x037490 to 0x01fced (0x177a2 bytes) 000665 1758808445.162925 mapread 0x000cd3 thru 0x00dd0a (0xd038 bytes) 000666 1758808445.171368 mapread 0x019b4e thru 0x01fced (0x61a0 bytes) 000667 1758808445.172253 read 0x0080ce thru 0x016afb (0xea2e bytes) 000668 1758808445.172859 mapwrite 0x02832f thru 0x037ee3 (0xfbb5 bytes) 000670 1758808445.260130 MIRROR_EXTEND to 15 mirrors 000671 1758808445.614641 MIRROR_EXTEND to 16 mirrors 000674 1758808445.929579 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000677 1758808446.155000 trunc from 0x037ee4 to 0x03786d (0x676 bytes) 000680 1758808446.219021 write 0x017bd2 thru 0x0180cb (0x4fa bytes) 000683 1758808446.225897 read 0x023963 thru 0x024a51 (0x10ef bytes) 000685 1758808446.230573 trunc from 0x03786e to 0x02ec72 (0x8bfb bytes) 000688 1758808446.247303 write 0x0321ef thru 0x0397f7 (0x7609 bytes) 000690 1758808446.252238 MIRROR_RESYNC 15 mirrors 000692 1758808447.372514 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000694 1758808447.688508 mapwrite 0x017fae thru 0x01e573 (0x65c6 bytes) 000696 1758808447.802735 mapread 0x02ef33 thru 0x035f6b (0x7039 bytes) 000699 1758808447.813845 mapwrite 0x035c20 thru 0x036b19 (0xefa bytes) 000700 1758808447.838458 write 0x03477f thru 0x03c287 (0x7b09 bytes) 000701 1758808447.839736 mapwrite 0x022bea thru 0x0279ee (0x4e05 bytes) 000704 1758808447.879987 mapread 0x01307e thru 0x022fc4 (0xff47 bytes) 000705 1758808447.880983 mapread 0x00940d thru 0x0174fc (0xe0f0 bytes) 000706 1758808447.889248 mapwrite 0x013c84 thru 0x022999 (0xed16 bytes) 000707 1758808447.941537 MIRROR_SPLIT mirror 8 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000709 1758808448.108996 mapwrite 0x03a89c thru 0x03ffff (0x5764 bytes) 000711 1758808448.167309 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000715 1758808448.320821 mapwrite 0x008070 thru 0x0133df (0xb370 bytes) 000716 1758808448.368087 mapwrite 0x02c507 thru 0x03bfe5 (0xfadf bytes) 000717 1758808448.425260 MIRROR_EXTEND to 13 mirrors 000720 1758808448.752114 mapwrite 0x010d52 thru 0x01c6cc (0xb97b bytes) 000721 1758808448.868110 write 0x0256d2 thru 0x02effa (0x9929 bytes) 000728 1758808448.880620 read 0x03fb88 thru 0x03ffff (0x478 bytes) 000729 1758808448.886708 mapread 0x005896 thru 0x00f9a0 (0xa10b bytes) 000730 1758808448.895727 read 0x03d04e thru 0x03e604 (0x15b7 bytes) 000731 1758808448.905175 mapwrite 0x005652 thru 0x013935 (0xe2e4 bytes) 000733 1758808448.944414 trunc from 0x040000 to 0x014259 (0x2bda6 bytes) 000734 1758808448.970103 trunc from 0x01425a to 0x025f70 (0x11d17 bytes) 000737 1758808448.979766 write 0x0188d1 thru 0x021acb (0x91fb bytes) 000740 1758808448.989969 MIRROR_EXTEND to 14 mirrors 000741 1758808449.309365 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000742 1758808449.514148 read 0x016f48 thru 0x021718 (0xa7d1 bytes) 000743 1758808449.527491 mapwrite 0x012540 thru 0x01272e (0x1ef bytes) 000744 1758808449.609243 write 0x01e0d2 thru 0x027348 (0x9277 bytes) 000745 1758808449.614927 trunc from 0x027349 to 0x0023bb (0x24f8d bytes) 000746 1758808449.632854 MIRROR_EXTEND to 14 mirrors 000747 1758808449.951853 trunc from 0x0023bc to 0x030787 (0x2e3cc bytes) 000749 1758808450.000230 write 0x023c1d thru 0x02c411 (0x87f5 bytes) 000750 1758808450.007538 read 0x0069e2 thru 0x00bb51 (0x5170 bytes) 000752 1758808450.014640 trunc from 0x030788 to 0x01a780 (0x16007 bytes) 000754 1758808450.027942 trunc from 0x01a781 to 0x00173c (0x19044 bytes) 000756 1758808450.038167 read 0x001416 thru 0x00173c (0x327 bytes) 000757 1758808450.044290 write 0x00d10a thru 0x00d8a5 (0x79c bytes) 000763 1758808450.047659 mapread 0x00254a thru 0x005fbe (0x3a75 bytes) 000765 1758808450.058279 MIRROR_RESYNC 14 mirrors 000766 1758808450.938120 read 0x009ef8 thru 0x00d8a5 (0x39ae bytes) 000769 1758808450.943494 mapread 0x001b0d thru 0x00d8a5 (0xbd99 bytes) 000770 1758808450.958002 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000773 1758808451.189633 MIRROR_RESYNC 13 mirrors 000775 1758808451.354321 write 0x026737 thru 0x02d446 (0x6d10 bytes) 000776 1758808451.397282 write 0x038095 thru 0x03ef0d (0x6e79 bytes) 000778 1758808451.398948 trunc from 0x03ef0e to 0x002c35 (0x3c2d8 bytes) 000781 1758808451.428458 write 0x01a10e thru 0x01eb82 (0x4a75 bytes) 000782 1758808451.429643 mapwrite 0x02c07d thru 0x02ce4f (0xdd3 bytes) 000784 1758808451.479686 MIRROR_EXTEND to 14 mirrors 000785 1758808451.818666 MIRROR_RESYNC 14 mirrors 000787 1758808452.708316 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000788 1758808452.996140 mapread 0x01f5c9 thru 0x0252e3 (0x5d1b bytes) 000789 1758808453.012269 mapwrite 0x021d7a thru 0x023153 (0x13da bytes) 000790 1758808453.133972 MIRROR_RESYNC 13 mirrors 000791 1758808454.256380 mapwrite 0x03a0c1 thru 0x03ce99 (0x2dd9 bytes) 000792 1758808454.405841 mapread 0x03610b thru 0x0383e4 (0x22da bytes) 000793 1758808454.410561 MIRROR_RESYNC 13 mirrors 000794 1758808455.345539 read 0x00d2bd thru 0x01916c (0xbeb0 bytes) 000797 1758808455.350348 trunc from 0x03ce9a to 0x00f28c (0x2dc0d bytes) 000798 1758808455.401997 write 0x02d971 thru 0x038c18 (0xb2a8 bytes) 000799 1758808455.406636 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000800 1758808455.526994 MIRROR_RESYNC 12 mirrors 000801 1758808456.326342 write 0x029e84 thru 0x031323 (0x74a0 bytes) 000802 1758808456.359939 write 0x0384f2 thru 0x03ffff (0x7b0e bytes) 000803 1758808456.366014 mapread 0x02fb70 thru 0x03ad1a (0xb1ab bytes) 000805 1758808456.367346 mapread 0x005aef thru 0x010638 (0xab4a bytes) 000809 1758808456.374339 write 0x02896f thru 0x02d71e (0x4db0 bytes) 000810 1758808456.381713 mapwrite 0x00fac8 thru 0x0192b0 (0x97e9 bytes) 000811 1758808456.420395 write 0x005635 thru 0x008f6d (0x3939 bytes) 000812 1758808456.421607 mapwrite 0x03eef4 thru 0x03ffff (0x110c bytes) 000814 1758808456.452758 mapread 0x001020 thru 0x00668c (0x566d bytes) 000815 1758808456.453608 mapwrite 0x019ae8 thru 0x028c7c (0xf195 bytes) 000817 1758808456.490455 write 0x025640 thru 0x026741 (0x1102 bytes) 000818 1758808456.494999 MIRROR_EXTEND to 13 mirrors 000821 1758808456.810787 write 0x005629 thru 0x00e20a (0x8be2 bytes) 000822 1758808456.870776 mapwrite 0x013a43 thru 0x01638d (0x294b bytes) 000826 1758808456.918849 write 0x01db65 thru 0x0273ff (0x989b bytes) 000830 1758808456.934571 MIRROR_EXTEND to 14 mirrors 000831 1758808457.335627 write 0x02e33a thru 0x02f9b2 (0x1679 bytes) 000833 1758808457.400621 trunc from 0x040000 to 0x031a53 (0xe5ac bytes) 000834 1758808457.416866 read 0x007732 thru 0x01305d (0xb92c bytes) 000835 1758808457.432261 mapwrite 0x036ac3 thru 0x0370d2 (0x610 bytes) 000836 1758808457.512978 mapread 0x021ec7 thru 0x02b017 (0x9151 bytes) 000837 1758808457.528932 read 0x00f340 thru 0x01dc81 (0xe942 bytes) 000839 1758808457.534828 write 0x01fb58 thru 0x0230cf (0x3578 bytes) 000840 1758808457.535842 read 0x0082a3 thru 0x00ba87 (0x37e5 bytes) 000841 1758808457.537451 read 0x025eff thru 0x0359f9 (0xfafb bytes) 000846 1758808457.540245 write 0x021a64 thru 0x02a1b9 (0x8756 bytes) 000849 1758808457.547772 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000852 1758808457.735760 mapread 0x01cfe9 thru 0x024519 (0x7531 bytes) 000855 1758808457.752162 mapread 0x01410a thru 0x022ec9 (0xedc0 bytes) 000857 1758808457.761460 read 0x01fb2d thru 0x028ecd (0x93a1 bytes) 000858 1758808457.766125 read 0x02e767 thru 0x02f01e (0x8b8 bytes) 000859 1758808457.774065 read 0x019946 thru 0x01b0d6 (0x1791 bytes) 000863 1758808457.780086 MIRROR_EXTEND to 14 mirrors 000867 1758808458.168122 mapwrite 0x0258a6 thru 0x02c19b (0x68f6 bytes) 000868 1758808458.289052 write 0x021b37 thru 0x02d72e (0xbbf8 bytes) 000869 1758808458.290196 write 0x02d056 thru 0x03bd2e (0xecd9 bytes) 000871 1758808458.291958 read 0x00bd6f thru 0x0162d8 (0xa56a bytes) 000872 1758808458.301034 mapwrite 0x009a41 thru 0x01563c (0xbbfc bytes) 000873 1758808458.345029 trunc from 0x03bd2f to 0x0121fa (0x29b34 bytes) 000875 1758808458.367009 read 0x00453c thru 0x0070fe (0x2bc3 bytes) 000880 1758808458.370083 mapread 0x000525 thru 0x008224 (0x7d00 bytes) 000883 1758808458.377534 read 0x011564 thru 0x0121fa (0xc97 bytes) 000885 1758808458.378528 read 0x00d2da thru 0x0121fa (0x4f21 bytes) 000887 1758808458.379378 read 0x004d9e thru 0x011441 (0xc6a4 bytes) 000888 1758808458.379904 write 0x00e672 thru 0x01d4d8 (0xee67 bytes) 000890 1758808458.381678 write 0x015e86 thru 0x022c07 (0xcd82 bytes) 000893 1758808458.383825 write 0x020ba3 thru 0x02128d (0x6eb bytes) 000894 1758808458.384585 mapwrite 0x03d84b thru 0x03ffff (0x27b5 bytes) 000898 1758808458.436931 read 0x00965b thru 0x0184f1 (0xee97 bytes) 000899 1758808458.443040 MIRROR_EXTEND to 15 mirrors 000900 1758808458.795965 write 0x00c4b3 thru 0x01c306 (0xfe54 bytes) 000901 1758808458.850919 write 0x016ea4 thru 0x01f79a (0x88f7 bytes) 000902 1758808458.858234 write 0x018dca thru 0x01f937 (0x6b6e bytes) 000904 1758808458.859665 mapread 0x0057a8 thru 0x00650a (0xd63 bytes) 000906 1758808458.870044 MIRROR_RESYNC 15 mirrors 000910 1758808459.974071 mapwrite 0x01cc2a thru 0x0289c0 (0xbd97 bytes) 000911 1758808460.135535 mapwrite 0x039f0f thru 0x03ffff (0x60f1 bytes) 000912 1758808460.176363 mapread 0x004a8b thru 0x00d8fe (0x8e74 bytes) 000913 1758808460.190730 mapread 0x00feab thru 0x0103b9 (0x50f bytes) 000914 1758808460.191586 write 0x03c252 thru 0x03ffff (0x3dae bytes) 000916 1758808460.192996 trunc from 0x040000 to 0x00a467 (0x35b98 bytes) 000917 1758808460.206771 mapwrite 0x01fde3 thru 0x0235f9 (0x3817 bytes) 000918 1758808460.249568 mapread 0x0010b3 thru 0x00c616 (0xb564 bytes) 000920 1758808460.257883 mapread 0x0116d8 thru 0x01e910 (0xd239 bytes) 000922 1758808460.263128 trunc from 0x0235fa to 0x01ddc7 (0x5832 bytes) 000923 1758808460.273256 mapwrite 0x032849 thru 0x03ffff (0xd7b7 bytes) 000925 1758808460.331119 trunc from 0x040000 to 0x01ac54 (0x253ab bytes) 000927 1758808460.343264 read 0x00e6d3 thru 0x01156f (0x2e9d bytes) 000928 1758808460.343888 mapwrite 0x024181 thru 0x02eb86 (0xaa06 bytes) 000929 1758808460.406734 trunc from 0x02eb87 to 0x0131be (0x1b9c8 bytes) 000930 1758808460.420770 mapwrite 0x02289a thru 0x026a2d (0x4194 bytes) 000932 1758808460.464813 mapwrite 0x018bca thru 0x021f52 (0x9389 bytes) 000933 1758808460.498549 trunc from 0x026a2e to 0x00ecc7 (0x17d66 bytes) 000936 1758808460.511684 trunc from 0x00ecc8 to 0x014bc4 (0x5efd bytes) 000937 1758808460.527342 mapread 0x00785a thru 0x014bc4 (0xd36b bytes) 000938 1758808460.534176 read 0x004fc0 thru 0x008e12 (0x3e53 bytes) 000939 1758808460.537787 MIRROR_EXTEND to 16 mirrors 000942 1758808460.866061 trunc from 0x014bc5 to 0x0303b4 (0x1b7f0 bytes) 000943 1758808460.926587 MIRROR_SPLIT mirror 10 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000946 1758808461.062249 trunc from 0x0303b5 to 0x00c030 (0x24384 bytes) 000948 1758808461.071180 mapwrite 0x01b430 thru 0x0204f0 (0x50c1 bytes) 000949 1758808461.116876 trunc from 0x0204f1 to 0x00711e (0x193d2 bytes) 000950 1758808461.127613 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000951 1758808461.240490 write 0x024035 thru 0x033317 (0xf2e3 bytes) 000953 1758808461.242337 trunc from 0x033318 to 0x00c6d5 (0x26c42 bytes) 000956 1758808461.260606 MIRROR_EXTEND to 15 mirrors 000957 1758808461.566196 write 0x02b6db thru 0x02f098 (0x39be bytes) 000958 1758808461.608032 mapread 0x017c54 thru 0x01ec27 (0x6fd4 bytes) 000959 1758808461.622124 mapread 0x026787 thru 0x02f098 (0x8912 bytes) 000960 1758808461.629375 MIRROR_EXTEND to 16 mirrors 000961 1758808461.920725 trunc from 0x02f099 to 0x01f8b5 (0xf7e3 bytes) 000962 1758808461.985087 MIRROR_SPLIT mirror 4 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3ed:0x0]: Invalid argument (22) 000965 1758808462.095992 MIRROR_EXTEND to 16 mirrors 000966 1758808462.426567 mapwrite 0x013166 thru 0x01cbbc (0x9a57 bytes) 000967 1758808462.530092 trunc from 0x01f8b6 to 0x0149b3 (0xaf02 bytes) 000968 1758808462.545017 read 0x00833a thru 0x0149b3 (0xc67a bytes) 000970 1758808462.555115 trunc from 0x0149b4 to 0x00868a (0xc329 bytes) 000972 1758808462.571578 read 0x004ac0 thru 0x006be0 (0x2121 bytes) 000976 1758808462.581040 write 0x02ae96 thru 0x036def (0xbf5a bytes) 000977 1758808462.583320 mapread 0x02a028 thru 0x036def (0xcdc8 bytes) 000979 1758808462.585288 read 0x006b23 thru 0x011345 (0xa823 bytes) 000980 1758808462.592188 trunc from 0x036df0 to 0x00e72d (0x286c2 bytes) 000984 1758808462.605200 read 0x00b64b thru 0x00e72d (0x30e3 bytes) 000986 1758808462.606368 write 0x015455 thru 0x01789b (0x2447 bytes) 000989 1758808462.608999 mapread 0x00508f thru 0x00f456 (0xa3c8 bytes) 000990 1758808462.615410 read 0x0096a8 thru 0x00f808 (0x6161 bytes) 000991 1758808462.615982 write 0x03d71c thru 0x03ffff (0x28e4 bytes) 000992 1758808462.616929 write 0x02f6a3 thru 0x03bb8e (0xc4ec bytes) 000993 1758808462.627282 trunc from 0x040000 to 0x015b0b (0x2a4f4 bytes) 000994 1758808462.646102 MIRROR_RESYNC 16 mirrors 000995 1758808463.782750 mapwrite 0x002546 thru 0x002d7d (0x838 bytes) 000996 1758808463.859262 mapwrite 0x022143 thru 0x02311d (0xfdb bytes) 000997 1758808463.899114 mapread 0x0077fb thru 0x00fc4d (0x8453 bytes) 000998 1758808463.904382 mapwrite 0x0137e5 thru 0x0171b5 (0x39d1 bytes) 000999 1758808463.935527 mapwrite 0x03fc98 thru 0x03ffff (0x368 bytes) 001000 1758808463.976474 mapread 0x033b5c thru 0x034d11 (0x11b6 bytes) All operations completed A-OK! PASS 70a (53s) == sanity-flr test 71: check mirror extend parent fid ==== 09:54:26 (1758808466) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 200a: stress test ===================== 09:54:27 (1758808467) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6864224 .. Extending file size to 2195936 .. failed Extending file size to 6279008 .. Extending file size to 7007616 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8722816 .. Extending file size to 8834208 .. Extending file size to 7079296 .. Extending file size to 7657600 .. Extending file size to 7986016 .. Extending file size to 8170560 .. Extending file size to 8833472 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8909728 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8894944 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9011072 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56104 56105 56106 56108 56111 done 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) 192.168.202.144@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (68s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:55:36 (1758808536) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6789920 .. Extending file size to 3806816 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5611680 .. Extending file size to 8552288 .. Extending file size to 8225184 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8617952 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8883968 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9305056 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8904928 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9202560 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58981 58982 58984 58986 58987 done 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) 192.168.202.144@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:56:42 (1758808602) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4608 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 4608 2203648 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0507024 s, 41.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114887 s, 18.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:56:50 (1758808610) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0348267 s, 60.2 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 09:56:52 (1758808612) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x43a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x583:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x583:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x584:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x586:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:56:55 (1758808615) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0392181 s, 53.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x585:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x587:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x588:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:56:57 (1758808617) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0583558 s, 35.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x586:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x587:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x589:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x586:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x587:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x589:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58a:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:57:00 (1758808620) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0370201 s, 56.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x588:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x588:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58c:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:57:25 (1758808645) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg244-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273685 s, 38.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x589:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x589:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58a:0x0] } PASS 204d (44s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:58:09 (1758808689) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg244-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0427085 s, 49.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x58f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:58:56 (1758808736) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg244-server mds-ost sync done. Creating new pool test_204f oleg244-server: Pool lustre.test_204f created Adding targets to pool oleg244-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg244-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.033749 s, 62.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x590:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x58d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x590:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg244-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg244-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg244-server: Pool lustre.test_204f destroyed PASS 204f (59s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:59:55 (1758808795) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:59:58 (1758808798) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:00:00 (1758808800) oleg244-server: Pool lustre.test_206 created Adding targets to pool oleg244-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg244-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg244-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg244-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg244-server: Pool lustre.test_206 destroyed PASS 206 (13s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:00:13 (1758808813) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261666 s, 40.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129696 s, 80.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0xe434b46e Mirror 3: 0xe434b46e PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:00:16 (1758808816) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:00:18 (1758808818) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:00:19 (1758808819) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0225812 s, 0.2 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x8c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xfa0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xfa0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xfa0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1640000 Read this (4 bytes): BBBB 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:00:35 (1758808835) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00562698 s, 1.5 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:00:37 (1758808837) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160044 s, 65.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x45f lmm_fid: [0x200000404:0x45f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x45f lmm_fid: [0x200000404:0x45f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x59a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835039 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-09-25 10:00:38.000000000 -0400 Modify: 2025-09-25 10:00:38.000000000 -0400 Change: 2025-09-25 10:00:38.000000000 -0400 Birth: 2025-09-25 10:00:37.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:00:40 (1758808840) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:00:41 (1758808841) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.143728 s, 285 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0166404 s, 246 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 25 10:00 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x461:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 25 10:00 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test complete, duration 909 sec ============ 10:00:44 (1758808844) === sanity-flr: start cleanup 10:00:44 (1758808844) === === sanity-flr: finish cleanup 10:00:46 (1758808846) ===