-----============= acceptance-small: sanity-flr ============----- Thu Jan 1 21:14:44 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg220-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:15:02 (1767320102) === oleg220-client.virtnet: executing check_config_client /mnt/lustre oleg220-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg220-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff95f010d1d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff95f010d1d800.idle_timeout=debug disable quota as required oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:15:52 (1767320152) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f9341] == sanity-flr test 0a: lfs mirror create with -N option == 21:15:54 (1767320154) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:16:08 (1767320168) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:16:10 (1767320170) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:16:13 (1767320173) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:16:35 (1767320195) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:16:38 (1767320198) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 21:16:42 (1767320202) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff95f010d1d800.stats=clear osc.lustre-OST0001-osc-ffff95f010d1d800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:16:48 (1767320208) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (8s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:16:56 (1767320216) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:17:04 (1767320224) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:17:06 (1767320226) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 21:17:14 (1767320234) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:17:21 (1767320241) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:17:25 (1767320245) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:17:32 (1767320252) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:17:42 (1767320262) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:17:48 (1767320268) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.566215 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138912 s, 7.5 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:17:55 (1767320275) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.285302 s, 44.1 MB/s PASS 22 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:18:02 (1767320282) Stopping /mnt/lustre-ost1 (opts:) on oleg220-server oleg220-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95f010d1d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95f010d1d800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg220-server: oleg220-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg220-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg220-server oleg220-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95f010d1d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95f010d1d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg220-server: oleg220-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg220-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (67s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:19:09 (1767320349) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.234379 s, 22.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg220-server oleg220-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95f010d1d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95f010d1d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg220-server: oleg220-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg220-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:19:51 (1767320391) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg220-server oleg220-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95f010d1d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95f010d1d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg220-server: oleg220-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg220-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg220-server oleg220-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95f010d1d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95f010d1d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg220-server: oleg220-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg220-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:21:05 (1767320465) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.97072 s, 13.2 MB/s -rw-r--r-- 1 root root 52428800 Jan 1 21:21 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.814s user 0m0.000s sys 0m0.177s Stopping /mnt/lustre-ost1 (opts:) on oleg220-server oleg220-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95f010d1d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95f010d1d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.352s user 0m0.002s sys 0m0.190s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg220-server: oleg220-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg220-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg220-server oleg220-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95f010d1d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95f010d1d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.415s user 0m0.001s sys 0m0.170s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg220-server: oleg220-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg220-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg220-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95f010d1d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95f010d1d800.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:22:38 (1767320558) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:22:42 (1767320562) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:22:45 (1767320565) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:22:48 (1767320568) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 21:22:56 (1767320576) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.623346 s, 20.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:23:08 (1767320588) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327337 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268549 s, 39.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:23:15 (1767320595) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:23:23 (1767320603) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00131054 s, 781 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00215438 s, 475 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00168125 s, 609 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0015062 s, 680 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 21:23:33 (1767320613) create test files with size 15729439 .. dc4bfb84c81e9e7b9bf2ac2eef724d7a - 830f92cc99c870f2a96310ef7f4ccb8b - 7fcbe21648cdb308a6240c45894c1f94 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 21:24:01 (1767320641) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:24:18 (1767320658) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:24:24 (1767320664) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071155 s, 14.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259399 s, 40.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 21:24:31 (1767320671) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151711 s, 13.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.269464 s, 15.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.302544 s, 6.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 21:24:47 (1767320687) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 21:24:50 (1767320690) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:24:53 (1767320693) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398146 s, 26.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 21:25:01 (1767320701) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:25:04 (1767320704) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:25:14 (1767320714) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.241713 s, 43.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:25:27 (1767320727) llite.lustre-ffff95f003812000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.345141 s, 30.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg220-client.virtnet /mnt/lustre (opts:) Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff95f00926a000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:25:40 (1767320740) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (33s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:26:13 (1767320773) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:26:46 (1767320806) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 21:26:49 (1767320809) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.212599 s, 14.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.222107 s, 14.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x40464206 (stale) Mirror 2: 0x40464206 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: 5bab1f89da02b3194a843f8e669da043 - mirror 1 checksum: c7815363e3fd5f968d5decf9ac1d0c87 - mirror 2 checksum: 5bab1f89da02b3194a843f8e669da043 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 1 21:26:53 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jan 1 21:26:53 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5bab1f89da02b3194a843f8e669da043 - mirror 1 checksum: c7815363e3fd5f968d5decf9ac1d0c87 - mirror 2 checksum: c7815363e3fd5f968d5decf9ac1d0c87 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 1 21:26:53 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jan 1 21:26:53 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:27:02 (1767320822) Failing mds1 on oleg220-server Stopping /mnt/lustre-mds1 (opts:) on oleg220-server 21:27:10 (1767320830) shut down facet: mds1 facet_host: oleg220-server facet_failover_host: oleg220-server Failover mds1 to oleg220-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg220-server: oleg220-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-MDT0000 21:27:46 (1767320866) targets are mounted 21:27:46 (1767320866) facet_failover done oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446016 s, 23.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 50A (62s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:28:05 (1767320885) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:28:08 (1767320888) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:28:11 (1767320891) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00612571 s, 2.7 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:28:18 (1767320898) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:28:21 (1767320901) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:28:25 (1767320905) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:28:48 (1767320928) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:29:17 (1767320957) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:29:41 (1767320981) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:30:05 (1767321005) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 21:30:12 (1767321012) mirror create pid 54756 mirror split pid 54757 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4952: 54756 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4952: 54757 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 21:31:19 (1767321079) Seed set to 1081 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1767321081.439123 write 0x03e9d4 thru 0x03ffff (0x162c bytes) 000008 1767321081.444443 mapwrite 0x019a4c thru 0x01b962 (0x1f17 bytes) 000009 1767321081.589338 mapread 0x01dffc thru 0x027fe2 (0x9fe7 bytes) 000010 1767321081.598919 write 0x0161cb thru 0x022295 (0xc0cb bytes) 000011 1767321081.600663 mapread 0x02360c thru 0x0288b0 (0x52a5 bytes) truncating to largest ever: 0x3eb8d 000013 1767321081.603073 trunc from 0x040000 to 0x03eb8c (0x1473 bytes) 000014 1767321081.636370 MIRROR_EXTEND to 2 mirrors 000016 1767321082.231158 write 0x02b860 thru 0x03109b (0x583c bytes) 000017 1767321082.365740 mapwrite 0x003dd0 thru 0x007ee2 (0x4113 bytes) 000018 1767321082.436378 read 0x024065 thru 0x033233 (0xf1cf bytes) 000021 1767321082.448214 mapwrite 0x031efb thru 0x03a6cd (0x87d3 bytes) 000022 1767321082.515040 read 0x00b7bf thru 0x017598 (0xbdda bytes) 000024 1767321082.560030 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000027 1767321082.846168 read 0x033b37 thru 0x03890b (0x4dd5 bytes) 000028 1767321082.881436 trunc from 0x03eb8d to 0x03bf82 (0x2c0a bytes) 000029 1767321082.948652 write 0x038ba2 thru 0x03ffff (0x745e bytes) 000030 1767321082.949853 write 0x006ef6 thru 0x00c511 (0x561c bytes) 000033 1767321082.969082 mapwrite 0x0361e6 thru 0x03ffff (0x9e1a bytes) 000038 1767321083.041339 mapwrite 0x02dc29 thru 0x039960 (0xbd38 bytes) 000039 1767321083.128576 write 0x024a5e thru 0x025421 (0x9c4 bytes) 000040 1767321083.137370 write 0x01f24e thru 0x027534 (0x82e7 bytes) 000041 1767321083.138473 read 0x01df87 thru 0x02d7de (0xf858 bytes) 000042 1767321083.141184 mapread 0x007a9e thru 0x0179b8 (0xff1b bytes) 000043 1767321083.152452 trunc from 0x040000 to 0x0304d3 (0xfb2c bytes) 000045 1767321083.198332 mapread 0x024b27 thru 0x028067 (0x3541 bytes) 000046 1767321083.199195 write 0x020dcd thru 0x0269a7 (0x5bdb bytes) 000047 1767321083.201060 write 0x02d1ee thru 0x0343ad (0x71c0 bytes) 000049 1767321083.203802 write 0x008804 thru 0x01737b (0xeb78 bytes) 000052 1767321083.217845 trunc from 0x0343ae to 0x00f72c (0x24c81 bytes) 000053 1767321083.256511 trunc from 0x00f72d to 0x00124a (0xe4e2 bytes) 000055 1767321083.283116 mapwrite 0x00a1b4 thru 0x012462 (0x82af bytes) 000057 1767321083.459381 write 0x02c5da thru 0x03651e (0x9f45 bytes) 000058 1767321083.464353 MIRROR_EXTEND to 2 mirrors 000060 1767321084.232650 trunc from 0x03651f to 0x0233e6 (0x13138 bytes) 000061 1767321084.401049 trunc from 0x0233e7 to 0x03104b (0xdc65 bytes) 000064 1767321084.437357 read 0x013eb0 thru 0x01c0b6 (0x8207 bytes) 000067 1767321084.451975 write 0x00389c thru 0x00e23a (0xa99f bytes) 000068 1767321084.463902 read 0x01acf5 thru 0x023ed7 (0x91e3 bytes) 000071 1767321084.475934 mapread 0x01a5de thru 0x01f9d2 (0x53f5 bytes) 000072 1767321084.476935 mapread 0x01c555 thru 0x029a04 (0xd4b0 bytes) 000079 1767321084.488150 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3104c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3104c): Mirror 2: 0x7a53b5f0 (stale) CRC-32 checksum value for chunk [0, 0x3104c): Mirror 1: 0xb7535a64 Mirror 3: 0xb7535a64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3104c: skipped 000082 1767321085.308488 read 0x017f86 thru 0x01f2f2 (0x736d bytes) 000083 1767321085.320112 write 0x00082b thru 0x002d11 (0x24e7 bytes) 000084 1767321085.441283 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3104c): Mirror 3: 0xb7535a64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3104c): Mirror 3: 0xb7535a64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3104c: skipped 000085 1767321085.777827 mapwrite 0x020e93 thru 0x02fe02 (0xef70 bytes) 000086 1767321085.956903 trunc from 0x03104c to 0x01d7ef (0x1385c bytes) 000088 1767321086.056832 mapwrite 0x03fabb thru 0x03ffff (0x545 bytes) 000091 1767321086.206211 MIRROR_RESYNC 2 mirrors 000098 1767321087.254953 mapread 0x011c14 thru 0x021565 (0xf952 bytes) 000101 1767321087.323898 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000102 1767321087.455622 write 0x01226c thru 0x017471 (0x5206 bytes) 000104 1767321087.499395 mapwrite 0x02e889 thru 0x039bac (0xb324 bytes) 000108 1767321087.597989 read 0x02dbe6 thru 0x032c16 (0x5031 bytes) 000110 1767321087.606515 MIRROR_EXTEND to 2 mirrors 000112 1767321088.523119 trunc from 0x040000 to 0x015c37 (0x2a3c8 bytes) 000113 1767321088.607295 trunc from 0x015c38 to 0x035b20 (0x1fee9 bytes) 000114 1767321088.633406 write 0x0337ba thru 0x03c742 (0x8f89 bytes) 000116 1767321088.643562 read 0x01e77a thru 0x028ca2 (0xa529 bytes) 000117 1767321088.656715 read 0x02081b thru 0x02b3d2 (0xabb8 bytes) 000118 1767321088.680979 trunc from 0x03c743 to 0x033c8f (0x8ab3 bytes) 000124 1767321088.720391 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000125 1767321088.934738 mapwrite 0x0323ed thru 0x034bdf (0x27f3 bytes) 000126 1767321089.023535 trunc from 0x034be0 to 0x000f99 (0x33c46 bytes) 000127 1767321089.054282 trunc from 0x000f9a to 0x02294e (0x219b5 bytes) 000128 1767321089.080311 write 0x023e30 thru 0x0305b2 (0xc783 bytes) 000130 1767321089.085524 mapwrite 0x032c8b thru 0x03ffff (0xd375 bytes) 000131 1767321089.177139 mapwrite 0x004418 thru 0x009fbe (0x5ba7 bytes) 000135 1767321089.236289 mapread 0x02a7f0 thru 0x033e46 (0x9657 bytes) 000136 1767321089.238973 mapread 0x0243f9 thru 0x02f7e2 (0xb3ea bytes) 000138 1767321089.241396 write 0x02d853 thru 0x039792 (0xbf40 bytes) 000140 1767321089.245485 mapread 0x037aa9 thru 0x0396bf (0x1c17 bytes) 000142 1767321089.247782 write 0x03dcf0 thru 0x03ffff (0x2310 bytes) 000147 1767321089.253063 trunc from 0x040000 to 0x0133a2 (0x2cc5d bytes) 000153 1767321089.282481 mapread 0x0112ac thru 0x0133a2 (0x20f7 bytes) 000154 1767321089.291396 trunc from 0x0133a3 to 0x023e56 (0x10ab4 bytes) 000156 1767321089.327897 write 0x02883f thru 0x02b938 (0x30fa bytes) 000157 1767321089.330446 mapwrite 0x001a86 thru 0x0073cb (0x5946 bytes) 000158 1767321089.435942 trunc from 0x02b939 to 0x011f77 (0x199c1 bytes) 000159 1767321089.489241 MIRROR_EXTEND to 2 mirrors 000160 1767321090.314927 MIRROR_RESYNC 2 mirrors 000161 1767321090.802322 MIRROR_RESYNC 2 mirrors 000163 1767321091.259739 MIRROR_EXTEND to 3 mirrors 000164 1767321092.426764 read 0x00d758 thru 0x011f77 (0x4820 bytes) 000165 1767321092.458187 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000169 1767321092.762655 mapread 0x0081e3 thru 0x00ecb5 (0x6ad3 bytes) 000170 1767321092.787063 read 0x007d28 thru 0x011f77 (0xa250 bytes) 000172 1767321092.792275 MIRROR_EXTEND to 3 mirrors 000175 1767321093.814641 mapread 0x007991 thru 0x00b25e (0x38ce bytes) 000176 1767321093.848474 read 0x003074 thru 0x00f5ba (0xc547 bytes) 000179 1767321093.894591 mapread 0x00ac97 thru 0x00d80e (0x2b78 bytes) 000181 1767321093.951241 mapwrite 0x007b21 thru 0x01488d (0xcd6d bytes) 000182 1767321094.150889 mapwrite 0x035bd9 thru 0x03fa95 (0x9ebd bytes) 000183 1767321094.241326 read 0x0289a7 thru 0x0310f6 (0x8750 bytes) 000184 1767321094.251543 write 0x015aab thru 0x01d712 (0x7c68 bytes) 000186 1767321094.255307 write 0x032a7b thru 0x03ffff (0xd585 bytes) 000187 1767321094.256739 mapwrite 0x02cede thru 0x0367fe (0x9921 bytes) 000191 1767321094.317457 trunc from 0x040000 to 0x006d6a (0x39295 bytes) 000192 1767321094.337433 trunc from 0x006d6b to 0x006312 (0xa58 bytes) 000193 1767321094.373345 trunc from 0x006313 to 0x01822a (0x11f18 bytes) 000195 1767321094.392282 trunc from 0x01822b to 0x009818 (0xea12 bytes) 000198 1767321094.408196 read 0x00712e thru 0x009818 (0x26eb bytes) 000200 1767321094.415283 mapwrite 0x00a2ac thru 0x01874e (0xe4a3 bytes) 000201 1767321094.503194 trunc from 0x01874f to 0x021d62 (0x9614 bytes) 000202 1767321094.526373 mapread 0x0180f5 thru 0x021d62 (0x9c6e bytes) 000203 1767321094.535401 write 0x02e9cc thru 0x03ae97 (0xc4cc bytes) 000204 1767321094.537894 mapread 0x026821 thru 0x0299cf (0x31af bytes) 000206 1767321094.548770 trunc from 0x03ae98 to 0x018164 (0x22d33 bytes) 000208 1767321094.605251 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18165): Mirror 4: 0xd5b00d94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18165): Mirror 4: 0xd5b00d94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18165: skipped 000209 1767321094.901502 mapwrite 0x00814f thru 0x00c43f (0x42f1 bytes) 000211 1767321095.063364 trunc from 0x018165 to 0x026ce2 (0xeb7e bytes) 000214 1767321095.091942 MIRROR_RESYNC 2 mirrors 000216 1767321095.821610 mapread 0x00c0b2 thru 0x00d503 (0x1452 bytes) 000219 1767321095.861646 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000220 1767321096.033469 mapread 0x013489 thru 0x018553 (0x50cb bytes) 000223 1767321096.063914 mapwrite 0x004b7d thru 0x006fde (0x2462 bytes) 000224 1767321096.144679 write 0x02b7f5 thru 0x032fe8 (0x77f4 bytes) 000226 1767321096.146649 mapwrite 0x01ef70 thru 0x0255d0 (0x6661 bytes) 000227 1767321096.251125 read 0x01042f thru 0x015e6d (0x5a3f bytes) 000228 1767321096.281221 trunc from 0x032fe9 to 0x02d1b2 (0x5e36 bytes) 000229 1767321096.310445 write 0x03dd6f thru 0x03ffff (0x2291 bytes) 000230 1767321096.314219 write 0x02b523 thru 0x03474c (0x922a bytes) 000231 1767321096.321666 write 0x01d6f1 thru 0x024eac (0x77bc bytes) 000232 1767321096.329220 MIRROR_EXTEND to 2 mirrors 000234 1767321097.095547 mapwrite 0x004113 thru 0x010951 (0xc83f bytes) 000235 1767321097.436495 mapread 0x0378b9 thru 0x038d60 (0x14a8 bytes) 000236 1767321097.453304 mapwrite 0x000232 thru 0x0074b0 (0x727f bytes) 000237 1767321097.536961 mapwrite 0x00777c thru 0x00d27c (0x5b01 bytes) 000238 1767321097.620168 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa89874ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x97040b14 Mirror 3: 0x97040b14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000240 1767321098.835643 read 0x00ab62 thru 0x00ec0d (0x40ac bytes) 000241 1767321098.852984 write 0x01fadc thru 0x024c16 (0x513b bytes) 000244 1767321098.947013 write 0x021ce4 thru 0x02cd39 (0xb056 bytes) 000246 1767321098.961014 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa89874ea (stale) Mirror 3: 0x97040b14 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2bbfcc93 Mirror 4: 0x2bbfcc93 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000248 1767321099.936125 read 0x027014 thru 0x034efe (0xdeeb bytes) 000250 1767321099.949020 mapwrite 0x02fa82 thru 0x0390ef (0x966e bytes) 000251 1767321100.123362 read 0x01198f thru 0x01db68 (0xc1da bytes) 000252 1767321100.141222 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa89874ea (stale) Mirror 4: 0x2bbfcc93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa89874ea (stale) Mirror 4: 0x2bbfcc93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000254 1767321100.553250 trunc from 0x040000 to 0x02799f (0x18660 bytes) 000255 1767321100.585661 mapread 0x022ab1 thru 0x02799f (0x4eef bytes) 000258 1767321100.613211 mapread 0x0123c5 thru 0x01e90d (0xc549 bytes) 000259 1767321100.636365 mapwrite 0x035567 thru 0x03b7fe (0x6298 bytes) 000260 1767321100.752445 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b7ff): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7ff): Mirror 2: 0xa851d9bd (stale) Mirror 4: 0xee4b586b (stale) CRC-32 checksum value for chunk [0, 0x3b7ff): Mirror 1: 0x83733cd9 Mirror 5: 0x83733cd9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7ff: skipped 000262 1767321101.926575 trunc from 0x03b7ff to 0x0019bd (0x39e41 bytes) 000270 1767321102.040256 write 0x012a8e thru 0x0216d9 (0xec4c bytes) 000271 1767321102.044998 read 0x013517 thru 0x018ffc (0x5ae6 bytes) 000273 1767321102.046370 mapread 0x00e5ae thru 0x01d9bb (0xf40e bytes) 000274 1767321102.070976 read 0x00b729 thru 0x014ebf (0x9797 bytes) 000275 1767321102.072114 mapwrite 0x022a45 thru 0x028b0a (0x60c6 bytes) 000276 1767321102.173593 read 0x00d4b0 thru 0x018653 (0xb1a4 bytes) 000277 1767321102.174236 trunc from 0x028b0b to 0x018f24 (0xfbe6 bytes) 000278 1767321102.196019 write 0x029b20 thru 0x0323e2 (0x88c3 bytes) 000279 1767321102.199148 mapread 0x022128 thru 0x02e7d9 (0xc6b2 bytes) 000280 1767321102.216007 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x323e3): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x323e3): Mirror 2: 0x3bbb85f9 (stale) Mirror 4: 0xa8445270 (stale) Mirror 5: 0xbad33c9a (stale) CRC-32 checksum value for chunk [0, 0x323e3): Mirror 1: 0xeb892a8 Mirror 6: 0xeb892a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x323e3: skipped 000282 1767321103.178569 read 0x029533 thru 0x0323e2 (0x8eb0 bytes) 000283 1767321103.194088 mapwrite 0x03e6aa thru 0x03fce3 (0x163a bytes) 000285 1767321103.513905 MIRROR_RESYNC 5 mirrors 000286 1767321105.098337 read 0x005b74 thru 0x00d0fa (0x7587 bytes) 000287 1767321105.117089 MIRROR_RESYNC 5 mirrors 000288 1767321105.744057 MIRROR_RESYNC 5 mirrors 000289 1767321106.429298 read 0x017a84 thru 0x0213d4 (0x9951 bytes) 000292 1767321106.457188 write 0x02975a thru 0x033659 (0x9f00 bytes) 000293 1767321106.548553 mapread 0x02b1f9 thru 0x02e7db (0x35e3 bytes) 000294 1767321106.549996 write 0x000dce thru 0x00fdfb (0xf02e bytes) 000296 1767321106.562546 trunc from 0x03fce4 to 0x018942 (0x273a1 bytes) 000297 1767321106.605325 trunc from 0x018943 to 0x02ac3a (0x122f8 bytes) 000298 1767321106.628606 trunc from 0x02ac3b to 0x02ef36 (0x42fc bytes) 000300 1767321106.646950 mapwrite 0x020453 thru 0x0279f0 (0x759e bytes) 000303 1767321106.699264 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ef37): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef37): Mirror 2: 0xe1dc6103 (stale) Mirror 4: 0xe1dc6103 (stale) Mirror 5: 0xe1dc6103 (stale) Mirror 6: 0xe1dc6103 (stale) CRC-32 checksum value for chunk [0, 0x2ef37): Mirror 1: 0x1c0e7cee Mirror 7: 0x1c0e7cee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef37: skipped 000304 1767321108.019474 mapread 0x0015ee thru 0x00b3ea (0x9dfd bytes) 000305 1767321108.064288 read 0x01b0b4 thru 0x028bae (0xdafb bytes) 000306 1767321108.086113 read 0x0147f2 thru 0x01ce87 (0x8696 bytes) 000308 1767321108.118248 trunc from 0x02ef37 to 0x00b4bf (0x23a77 bytes) 000309 1767321108.268308 mapread 0x0007da thru 0x00624d (0x5a74 bytes) 000310 1767321108.340012 read 0x006b9b thru 0x00b354 (0x47ba bytes) 000311 1767321108.349516 trunc from 0x00b4c0 to 0x03c8b5 (0x313f6 bytes) 000314 1767321108.415293 MIRROR_RESYNC 6 mirrors 000315 1767321109.759272 mapread 0x01965a thru 0x025a87 (0xc42e bytes) 000316 1767321109.794370 read 0x0172a0 thru 0x024969 (0xd6ca bytes) 000317 1767321109.794875 mapread 0x034736 thru 0x038e0b (0x46d6 bytes) 000318 1767321109.808547 read 0x0262a1 thru 0x02810d (0x1e6d bytes) 000319 1767321109.810193 mapread 0x03387a thru 0x03c8b5 (0x903c bytes) 000320 1767321109.812086 write 0x015b71 thru 0x025a44 (0xfed4 bytes) 000322 1767321109.921893 write 0x00a9e3 thru 0x0197e0 (0xedfe bytes) 000323 1767321109.925073 write 0x022208 thru 0x0289cf (0x67c8 bytes) 000324 1767321109.927458 mapwrite 0x00ec61 thru 0x01e1f9 (0xf599 bytes) 000325 1767321109.996745 read 0x02fe34 thru 0x0398cc (0x9a99 bytes) 000326 1767321109.998428 mapread 0x02cd6d thru 0x031e49 (0x50dd bytes) 000327 1767321109.999934 mapread 0x0210c7 thru 0x025e4a (0x4d84 bytes) 000328 1767321110.002318 mapwrite 0x024ee1 thru 0x025bef (0xd0f bytes) 000329 1767321110.051607 read 0x01d242 thru 0x022ee4 (0x5ca3 bytes) 000334 1767321110.056587 mapwrite 0x03cf95 thru 0x03ffff (0x306b bytes) 000335 1767321110.166743 read 0x028602 thru 0x0338d5 (0xb2d4 bytes) 000336 1767321110.167614 trunc from 0x040000 to 0x030c8a (0xf375 bytes) 000337 1767321110.189730 mapread 0x013791 thru 0x0151f2 (0x1a62 bytes) 000339 1767321110.191786 write 0x01ffcf thru 0x024eea (0x4f1c bytes) 000340 1767321110.194081 mapread 0x02daa6 thru 0x0304d8 (0x2a33 bytes) 000342 1767321110.197776 write 0x0079ae thru 0x00db9a (0x61ed bytes) 000345 1767321110.200860 write 0x00fc2e thru 0x013146 (0x3519 bytes) 000346 1767321110.207899 MIRROR_RESYNC 6 mirrors 000347 1767321111.312613 mapwrite 0x0357e6 thru 0x03ffff (0xa81a bytes) 000348 1767321111.540805 trunc from 0x040000 to 0x000766 (0x3f899 bytes) 000349 1767321111.584586 MIRROR_RESYNC 6 mirrors 000352 1767321112.650237 read 0x0006e5 thru 0x000766 (0x82 bytes) 000354 1767321112.657637 mapwrite 0x016aee thru 0x018fc2 (0x24d5 bytes) 000357 1767321112.841230 read 0x00373a thru 0x006699 (0x2f60 bytes) 000359 1767321112.853076 read 0x017ec9 thru 0x018fc2 (0x10fa bytes) 000360 1767321112.854492 mapread 0x002d41 thru 0x00e9b8 (0xbc78 bytes) 000361 1767321112.855747 mapread 0x006295 thru 0x01548b (0xf1f7 bytes) 000362 1767321112.858035 mapwrite 0x022148 thru 0x02da8a (0xb943 bytes) 000363 1767321112.949806 trunc from 0x02da8b to 0x017052 (0x16a38 bytes) 000364 1767321112.973208 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17053): Mirror 4: 0xb195c754 (stale) Mirror 5: 0xb195c754 (stale) Mirror 6: 0xb195c754 (stale) Mirror 7: 0xb195c754 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17053): Mirror 4: 0xb195c754 (stale) Mirror 5: 0xb195c754 (stale) Mirror 6: 0xb195c754 (stale) Mirror 7: 0xb195c754 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17053: skipped 000366 1767321113.348608 write 0x01119b thru 0x014917 (0x377d bytes) 000370 1767321113.362149 trunc from 0x017053 to 0x01e93a (0x78e8 bytes) 000371 1767321113.392304 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1e93b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e93b): Mirror 4: 0x20319c68 (stale) Mirror 5: 0x20319c68 (stale) Mirror 6: 0x20319c68 (stale) Mirror 7: 0x20319c68 (stale) CRC-32 checksum value for chunk [0, 0x1e93b): Mirror 1: 0xb5cf78f4 Mirror 8: 0xb5cf78f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e93b: skipped 000372 1767321114.385284 trunc from 0x01e93b to 0x01c185 (0x27b5 bytes) 000376 1767321114.486082 mapread 0x015db1 thru 0x01c185 (0x63d5 bytes) 000378 1767321114.505456 write 0x009b2c thru 0x00d286 (0x375b bytes) 000382 1767321114.517821 mapread 0x011deb thru 0x0159c1 (0x3bd7 bytes) 000384 1767321114.521204 mapwrite 0x01b756 thru 0x023d2b (0x85d6 bytes) 000385 1767321114.672865 mapread 0x01fbc3 thru 0x0219e8 (0x1e26 bytes) 000388 1767321114.680676 trunc from 0x023d2c to 0x035b5d (0x11e32 bytes) 000390 1767321114.715582 mapread 0x00e359 thru 0x01ae1a (0xcac2 bytes) 000391 1767321114.723532 trunc from 0x035b5e to 0x01f17a (0x169e3 bytes) 000394 1767321114.778498 trunc from 0x01f17b to 0x00f5b2 (0xfbc8 bytes) 000395 1767321114.811443 trunc from 0x00f5b3 to 0x00a4cf (0x50e3 bytes) 000397 1767321114.843058 MIRROR_RESYNC 6 mirrors 000398 1767321116.195833 read 0x008d16 thru 0x00a4cf (0x17ba bytes) 000401 1767321116.226551 write 0x01f7b5 thru 0x02b6a2 (0xbeee bytes) 000403 1767321116.341956 trunc from 0x02b6a3 to 0x031beb (0x6549 bytes) 000405 1767321116.365817 trunc from 0x031bec to 0x023912 (0xe2d9 bytes) 000406 1767321116.387482 write 0x033b66 thru 0x03ffff (0xc49a bytes) 000407 1767321116.389950 mapread 0x020411 thru 0x030113 (0xfd03 bytes) 000408 1767321116.436206 read 0x02be80 thru 0x0326eb (0x686c bytes) 000411 1767321116.439950 trunc from 0x040000 to 0x02f878 (0x10787 bytes) 000412 1767321116.468572 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f879): Mirror 5: 0xff2a0e64 (stale) Mirror 6: 0xff2a0e64 (stale) Mirror 7: 0xff2a0e64 (stale) Mirror 8: 0xff2a0e64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f879): Mirror 5: 0xff2a0e64 (stale) Mirror 6: 0xff2a0e64 (stale) Mirror 7: 0xff2a0e64 (stale) Mirror 8: 0xff2a0e64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f879: skipped 000413 1767321116.944485 mapwrite 0x016714 thru 0x018e99 (0x2786 bytes) 000415 1767321117.020666 write 0x02b8da thru 0x039c4e (0xe375 bytes) 000419 1767321117.037272 mapread 0x01e54f thru 0x01fc4d (0x16ff bytes) 000422 1767321117.039672 trunc from 0x039c4f to 0x001cff (0x37f4f bytes) 000423 1767321117.069490 trunc from 0x001d00 to 0x00a1d5 (0x84d6 bytes) 000429 1767321117.101074 read 0x003320 thru 0x00a1d5 (0x6eb6 bytes) 000432 1767321117.112098 trunc from 0x00a1d6 to 0x01e058 (0x13e83 bytes) 000434 1767321117.155444 mapwrite 0x03101f thru 0x03c205 (0xb1e7 bytes) 000436 1767321117.408304 MIRROR_RESYNC 5 mirrors 000437 1767321118.740069 mapread 0x012e89 thru 0x01a0ed (0x7265 bytes) 000438 1767321118.760068 write 0x02d4ff thru 0x03b00a (0xdb0c bytes) 000439 1767321118.839672 trunc from 0x03c206 to 0x0338fb (0x890a bytes) 000440 1767321118.864495 read 0x007689 thru 0x010add (0x9455 bytes) 000441 1767321118.882385 mapread 0x010a1a thru 0x01dbd0 (0xd1b7 bytes) 000444 1767321118.902357 read 0x008893 thru 0x00f2eb (0x6a59 bytes) 000445 1767321118.909067 mapwrite 0x01c74c thru 0x01fc05 (0x34ba bytes) 000446 1767321118.965180 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x338fc): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338fc): Mirror 5: 0x10c78a3f (stale) Mirror 6: 0x10c78a3f (stale) Mirror 7: 0x10c78a3f (stale) Mirror 8: 0x10c78a3f (stale) CRC-32 checksum value for chunk [0, 0x338fc): Mirror 1: 0x61e77c9 Mirror 9: 0x61e77c9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338fc: skipped 000447 1767321119.839246 mapwrite 0x002782 thru 0x00705c (0x48db bytes) 000448 1767321120.024207 mapwrite 0x012b11 thru 0x020ffa (0xe4ea bytes) 000449 1767321120.092202 write 0x028d7c thru 0x0293f7 (0x67c bytes) 000450 1767321120.106672 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x338fc): Mirror 5: 0x10c78a3f (stale) Mirror 6: 0x10c78a3f (stale) Mirror 7: 0x10c78a3f (stale) Mirror 8: 0x10c78a3f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338fc): Mirror 5: 0x10c78a3f (stale) Mirror 6: 0x10c78a3f (stale) Mirror 7: 0x10c78a3f (stale) Mirror 8: 0x10c78a3f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338fc: skipped 000451 1767321120.540403 write 0x038b2a thru 0x03ffff (0x74d6 bytes) 000452 1767321120.543445 read 0x00323c thru 0x011122 (0xdee7 bytes) 000454 1767321120.556567 mapread 0x01c9dc thru 0x020992 (0x3fb7 bytes) 000455 1767321120.576771 trunc from 0x040000 to 0x0075fb (0x38a04 bytes) 000459 1767321120.607030 mapwrite 0x008ebf thru 0x00d444 (0x4586 bytes) 000462 1767321120.690574 read 0x00499b thru 0x00d444 (0x8aaa bytes) 000464 1767321120.692534 mapread 0x00a897 thru 0x00b2e1 (0xa4b bytes) 000465 1767321120.693563 read 0x006fcb thru 0x00d444 (0x647a bytes) 000466 1767321120.694615 write 0x014c6b thru 0x015902 (0xc98 bytes) 000470 1767321120.702535 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x15903): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15903): Mirror 5: 0xcd8e74cb (stale) Mirror 6: 0xcd8e74cb (stale) Mirror 7: 0xcd8e74cb (stale) Mirror 8: 0xcd8e74cb (stale) CRC-32 checksum value for chunk [0, 0x15903): Mirror 1: 0xb38f80d0 Mirror 9: 0xb38f80d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15903: skipped 000472 1767321121.743598 mapread 0x006af0 thru 0x01028e (0x979f bytes) 000476 1767321121.814603 MIRROR_RESYNC 6 mirrors 000477 1767321123.059608 MIRROR_RESYNC 6 mirrors 000481 1767321123.751572 read 0x00c23b thru 0x014a59 (0x881f bytes) 000482 1767321123.776779 write 0x030c35 thru 0x03edb3 (0xe17f bytes) 000483 1767321123.919518 write 0x029a3c thru 0x02a245 (0x80a bytes) 000484 1767321123.967992 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3edb4): Mirror 6: 0x708d13a4 (stale) Mirror 7: 0x708d13a4 (stale) Mirror 8: 0x708d13a4 (stale) Mirror 9: 0x708d13a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3edb4): Mirror 6: 0x708d13a4 (stale) Mirror 7: 0x708d13a4 (stale) Mirror 8: 0x708d13a4 (stale) Mirror 9: 0x708d13a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3edb4: skipped 000485 1767321124.784056 mapread 0x01cb81 thru 0x01db1d (0xf9d bytes) 000486 1767321124.807022 read 0x00ba32 thru 0x0190ed (0xd6bc bytes) 000487 1767321124.838185 mapread 0x0095f7 thru 0x01279c (0x91a6 bytes) 000488 1767321124.865531 trunc from 0x03edb4 to 0x0248e9 (0x1a4ca bytes) 000489 1767321124.905771 mapwrite 0x039197 thru 0x03de50 (0x4cba bytes) 000492 1767321125.037834 write 0x02bca5 thru 0x03956f (0xd8cb bytes) 000493 1767321125.061449 read 0x00036a thru 0x00f2a5 (0xef3c bytes) 000495 1767321125.076718 write 0x02532d thru 0x02a24a (0x4f1e bytes) 000496 1767321125.078506 mapread 0x01b678 thru 0x01cbde (0x1567 bytes) 000497 1767321125.079950 write 0x0120b0 thru 0x016e3a (0x4d8b bytes) 000499 1767321125.081885 write 0x013f51 thru 0x0232a0 (0xf350 bytes) 000500 1767321125.085888 trunc from 0x03de51 to 0x038afb (0x5355 bytes) 000501 1767321125.119965 write 0x02d060 thru 0x03a4c0 (0xd461 bytes) 000502 1767321125.128737 trunc from 0x03a4c1 to 0x01f421 (0x1b09f bytes) 000503 1767321125.190893 mapwrite 0x038b21 thru 0x03ffff (0x74df bytes) 000505 1767321125.339433 MIRROR_RESYNC 5 mirrors 000506 1767321126.688973 read 0x023d13 thru 0x0300b8 (0xc3a6 bytes) 000508 1767321126.704063 trunc from 0x040000 to 0x030a36 (0xf5c9 bytes) 000509 1767321126.805345 read 0x02cdc3 thru 0x02d157 (0x395 bytes) 000511 1767321126.829883 write 0x0122cf thru 0x0166ca (0x43fc bytes) 000512 1767321126.831838 read 0x010f32 thru 0x015a1b (0x4aea bytes) 000513 1767321126.832840 mapread 0x00f0d9 thru 0x01ec50 (0xfb78 bytes) 000514 1767321126.834497 read 0x029d82 thru 0x02d310 (0x358f bytes) 000515 1767321126.835460 write 0x019308 thru 0x0271d6 (0xdecf bytes) 000516 1767321126.837523 trunc from 0x030a37 to 0x001e86 (0x2ebb0 bytes) 000518 1767321126.902624 write 0x01d028 thru 0x026173 (0x914c bytes) 000519 1767321126.938501 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x26174): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26174): Mirror 6: 0x26cc220 (stale) Mirror 7: 0x26cc220 (stale) Mirror 8: 0x26cc220 (stale) Mirror 9: 0x26cc220 (stale) CRC-32 checksum value for chunk [0, 0x26174): Mirror 1: 0x6f117fb3 Mirror 10: 0x6f117fb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26174: skipped 000520 1767321128.143710 mapread 0x00999c thru 0x00a868 (0xecd bytes) 000521 1767321128.177402 mapwrite 0x0047e3 thru 0x0082b9 (0x3ad7 bytes) 000522 1767321128.385556 write 0x02d309 thru 0x038032 (0xad2a bytes) 000525 1767321128.388420 write 0x0115df thru 0x01d973 (0xc395 bytes) 000526 1767321128.403265 write 0x018315 thru 0x0193ed (0x10d9 bytes) 000527 1767321128.404116 read 0x03777e thru 0x038032 (0x8b5 bytes) 000528 1767321128.404614 read 0x028115 thru 0x037bc1 (0xfaad bytes) 000529 1767321128.413260 read 0x00b0b9 thru 0x017a86 (0xc9ce bytes) 000532 1767321128.419657 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38033): Mirror 6: 0x4e95b9f9 (stale) Mirror 8: 0x4e95b9f9 (stale) Mirror 9: 0x4e95b9f9 (stale) Mirror 10: 0xbacaf9cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38033): Mirror 6: 0x4e95b9f9 (stale) Mirror 8: 0x4e95b9f9 (stale) Mirror 9: 0x4e95b9f9 (stale) Mirror 10: 0xbacaf9cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38033: skipped 000535 1767321128.794424 mapwrite 0x03da12 thru 0x03ffff (0x25ee bytes) 000537 1767321128.868349 write 0x0385f0 thru 0x03863f (0x50 bytes) 000540 1767321128.871667 mapread 0x031c98 thru 0x037774 (0x5add bytes) 000542 1767321128.881070 write 0x016774 thru 0x025bfd (0xf48a bytes) 000543 1767321128.895505 trunc from 0x040000 to 0x00f99c (0x30663 bytes) 000545 1767321128.922049 mapread 0x00d81e thru 0x00f99c (0x217f bytes) 000546 1767321128.932659 mapwrite 0x03da00 thru 0x03f630 (0x1c31 bytes) 000547 1767321129.008860 mapwrite 0x0240cd thru 0x02471e (0x652 bytes) 000548 1767321129.065087 trunc from 0x03f631 to 0x036688 (0x8fa8 bytes) 000549 1767321129.089647 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36689): Mirror 6: 0xf04a684 (stale) Mirror 9: 0xf04a684 (stale) Mirror 10: 0x54f0fb4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36689): Mirror 6: 0xf04a684 (stale) Mirror 9: 0xf04a684 (stale) Mirror 10: 0x54f0fb4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36689: skipped 000552 1767321129.467329 read 0x0050c2 thru 0x01442d (0xf36c bytes) 000554 1767321129.480658 trunc from 0x036689 to 0x0082d0 (0x2e3b8 bytes) 000555 1767321129.500500 trunc from 0x0082d1 to 0x0368e4 (0x2e614 bytes) 000556 1767321129.521591 trunc from 0x0368e5 to 0x01da60 (0x18e84 bytes) 000558 1767321129.548298 read 0x01b190 thru 0x01da60 (0x28d1 bytes) 000563 1767321129.565941 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1da61): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1da61): Mirror 6: 0xf7472b08 (stale) Mirror 9: 0xf7472b08 (stale) Mirror 10: 0x4fcfbb4b (stale) CRC-32 checksum value for chunk [0, 0x1da61): Mirror 1: 0xc8c728cc Mirror 11: 0xc8c728cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1da61: skipped 000565 1767321130.453313 mapread 0x01ce10 thru 0x01da60 (0xc51 bytes) 000566 1767321130.487579 trunc from 0x01da61 to 0x034fb8 (0x17558 bytes) 000567 1767321130.654667 read 0x01c6b8 thru 0x02770a (0xb053 bytes) 000568 1767321130.666535 write 0x030151 thru 0x037d2a (0x7bda bytes) 000570 1767321130.680134 trunc from 0x037d2b to 0x034791 (0x3599 bytes) 000571 1767321130.712499 read 0x009b16 thru 0x00e35b (0x4846 bytes) 000574 1767321130.728650 trunc from 0x034792 to 0x03afd8 (0x6847 bytes) 000575 1767321130.754400 mapread 0x01fc05 thru 0x020821 (0xc1d bytes) 000576 1767321130.756391 write 0x01fe7e thru 0x02b118 (0xb29b bytes) 000579 1767321130.766412 MIRROR_RESYNC 5 mirrors 000580 1767321132.122419 mapread 0x031df9 thru 0x03afd8 (0x91e0 bytes) 000582 1767321132.166942 mapwrite 0x032728 thru 0x03c684 (0x9f5d bytes) 000583 1767321132.379543 write 0x022dcc thru 0x02689e (0x3ad3 bytes) 000585 1767321132.415169 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c685): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c685): Mirror 6: 0x9ec70427 (stale) Mirror 9: 0x9ec70427 (stale) Mirror 10: 0x9ec70427 (stale) Mirror 11: 0x9ec70427 (stale) CRC-32 checksum value for chunk [0, 0x3c685): Mirror 1: 0xc4e8d8f Mirror 12: 0xc4e8d8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c685: skipped 000586 1767321134.045375 read 0x03145e thru 0x03c684 (0xb227 bytes) 000587 1767321134.065038 mapread 0x02cecb thru 0x02ea83 (0x1bb9 bytes) 000589 1767321134.112450 write 0x0322fd thru 0x036771 (0x4475 bytes) 000590 1767321134.178656 trunc from 0x03c685 to 0x006c2a (0x35a5a bytes) 000592 1767321134.200918 mapwrite 0x0267f3 thru 0x02be26 (0x5634 bytes) 000594 1767321134.275929 mapwrite 0x011054 thru 0x01b760 (0xa70d bytes) 000595 1767321134.388939 mapwrite 0x02db0b thru 0x03b259 (0xd74f bytes) 000596 1767321134.560296 mapread 0x008fd0 thru 0x018792 (0xf7c3 bytes) 000598 1767321134.577422 mapread 0x01715f thru 0x020d94 (0x9c36 bytes) 000601 1767321134.581110 mapread 0x0144cf thru 0x022600 (0xe132 bytes) 000602 1767321134.583101 mapread 0x011647 thru 0x015242 (0x3bfc bytes) 000604 1767321134.584546 write 0x02e96f thru 0x0369f2 (0x8084 bytes) 000610 1767321134.595424 MIRROR_RESYNC 6 mirrors 000613 1767321135.757878 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000614 1767321136.201465 mapread 0x003a34 thru 0x003b51 (0x11e bytes) 000615 1767321136.218932 mapread 0x00098a thru 0x00f02a (0xe6a1 bytes) 000616 1767321136.249939 write 0x0244db thru 0x02500d (0xb33 bytes) 000617 1767321136.428939 trunc from 0x03b25a to 0x02761c (0x13c3d bytes) 000618 1767321136.466750 read 0x009d97 thru 0x0161a9 (0xc413 bytes) 000620 1767321136.496008 mapwrite 0x032eb6 thru 0x03eaf1 (0xbc3c bytes) 000621 1767321136.690123 mapwrite 0x024f60 thru 0x02b45f (0x6500 bytes) 000622 1767321136.793402 trunc from 0x03eaf2 to 0x025507 (0x195ea bytes) 000623 1767321136.824243 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25508): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25508): Mirror 6: 0xf29593e8 (stale) Mirror 9: 0xf29593e8 (stale) Mirror 11: 0xf29593e8 (stale) Mirror 12: 0xf29593e8 (stale) CRC-32 checksum value for chunk [0, 0x25508): Mirror 1: 0xef196e05 Mirror 13: 0xef196e05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25508: skipped 000626 1767321138.278504 trunc from 0x025508 to 0x036643 (0x1113c bytes) 000629 1767321138.415547 read 0x029309 thru 0x0338dd (0xa5d5 bytes) 000630 1767321138.435578 mapread 0x00b486 thru 0x010b50 (0x56cb bytes) 000632 1767321138.472402 MIRROR_RESYNC 6 mirrors 000633 1767321139.903544 mapwrite 0x025291 thru 0x02e67a (0x93ea bytes) 000634 1767321140.161406 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36644): Mirror 6: 0x265dd34d (stale) Mirror 9: 0x265dd34d (stale) Mirror 11: 0x265dd34d (stale) Mirror 13: 0x265dd34d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36644): Mirror 6: 0x265dd34d (stale) Mirror 9: 0x265dd34d (stale) Mirror 11: 0x265dd34d (stale) Mirror 13: 0x265dd34d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36644: skipped 000635 1767321140.558350 write 0x03b67c thru 0x03ffff (0x4984 bytes) 000636 1767321140.559430 trunc from 0x040000 to 0x01b667 (0x24998 bytes) 000638 1767321140.585504 trunc from 0x01b668 to 0x013317 (0x8350 bytes) 000639 1767321140.608101 MIRROR_RESYNC 5 mirrors 000641 1767321141.545256 trunc from 0x013318 to 0x011eae (0x1469 bytes) 000642 1767321141.624025 read 0x00fba2 thru 0x011eae (0x230d bytes) 000643 1767321141.634924 mapwrite 0x017f2c thru 0x026d8d (0xee62 bytes) 000645 1767321141.753587 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26d8e): Mirror 6: 0x7d2223d7 (stale) Mirror 9: 0x7d2223d7 (stale) Mirror 11: 0x7d2223d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26d8e): Mirror 6: 0x7d2223d7 (stale) Mirror 9: 0x7d2223d7 (stale) Mirror 11: 0x7d2223d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26d8e: skipped 000647 1767321142.113099 write 0x0164c5 thru 0x0184a8 (0x1fe4 bytes) 000649 1767321142.126778 mapwrite 0x017e0b thru 0x023d42 (0xbf38 bytes) 000650 1767321142.224854 mapread 0x00a6b6 thru 0x01a1c7 (0xfb12 bytes) 000652 1767321142.237297 read 0x0155fc thru 0x01f63e (0xa043 bytes) 000654 1767321142.238434 trunc from 0x026d8e to 0x03b4f6 (0x14769 bytes) 000655 1767321142.258587 MIRROR_RESYNC 4 mirrors 000656 1767321143.167616 trunc from 0x03b4f7 to 0x003650 (0x37ea6 bytes) 000657 1767321143.274027 read 0x001a98 thru 0x003650 (0x1bb9 bytes) 000658 1767321143.288805 mapread 0x0010d5 thru 0x003650 (0x257c bytes) 000659 1767321143.290630 mapread 0x002d19 thru 0x003650 (0x938 bytes) 000660 1767321143.291806 read 0x000fc3 thru 0x003650 (0x268e bytes) 000662 1767321143.301202 read 0x0031b3 thru 0x003650 (0x49e bytes) 000664 1767321143.302529 mapwrite 0x02de24 thru 0x02f7ce (0x19ab bytes) 000668 1767321143.434609 mapread 0x00dd85 thru 0x01c375 (0xe5f1 bytes) 000669 1767321143.437211 trunc from 0x02f7cf to 0x01ecf8 (0x10ad6 bytes) 000671 1767321143.473346 write 0x00d82b thru 0x0127d8 (0x4fae bytes) 000672 1767321143.474749 trunc from 0x01ecf9 to 0x002c08 (0x1c0f0 bytes) 000674 1767321143.505840 read 0x0017a3 thru 0x002c08 (0x1466 bytes) 000675 1767321143.507434 read 0x00242f thru 0x002c08 (0x7da bytes) 000676 1767321143.508012 write 0x0293ef thru 0x030dec (0x79fe bytes) 000679 1767321143.520154 trunc from 0x030ded to 0x0276da (0x9712 bytes) 000680 1767321143.546994 mapwrite 0x00ea42 thru 0x019867 (0xae26 bytes) 000681 1767321143.635436 write 0x03151d thru 0x035514 (0x3ff8 bytes) 000682 1767321143.637720 trunc from 0x035515 to 0x002cd3 (0x32841 bytes) 000684 1767321143.663220 read 0x001ba2 thru 0x002cd3 (0x1132 bytes) 000686 1767321143.664344 trunc from 0x002cd4 to 0x0265a4 (0x238d1 bytes) 000687 1767321143.686381 trunc from 0x0265a5 to 0x023aa9 (0x2afb bytes) 000688 1767321143.701572 read 0x015f71 thru 0x023265 (0xd2f5 bytes) 000690 1767321143.715457 mapwrite 0x02272c thru 0x02e3ad (0xbc82 bytes) 000691 1767321143.812003 mapread 0x004096 thru 0x009a95 (0x5a00 bytes) 000694 1767321143.813892 mapread 0x02a73d thru 0x02d348 (0x2c0c bytes) 000696 1767321143.817171 trunc from 0x02e3ae to 0x01a32d (0x14080 bytes) 000700 1767321143.842397 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a32e): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a32e): Mirror 6: 0x71714a23 (stale) Mirror 9: 0x71714a23 (stale) Mirror 11: 0x71714a23 (stale) CRC-32 checksum value for chunk [0, 0x1a32e): Mirror 1: 0x392a2348 Mirror 12: 0x392a2348 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a32e: skipped 000701 1767321144.624446 read 0x00d2c2 thru 0x017b4c (0xa88b bytes) 000702 1767321144.649062 read 0x01928c thru 0x01a32d (0x10a2 bytes) 000703 1767321144.677931 read 0x00b282 thru 0x00fc0d (0x498c bytes) 000704 1767321144.691611 mapread 0x01320f thru 0x01a32d (0x711f bytes) 000705 1767321144.713220 write 0x01477f thru 0x0214a4 (0xcd26 bytes) 000706 1767321144.804517 read 0x01b053 thru 0x0214a4 (0x6452 bytes) 000709 1767321144.805561 trunc from 0x0214a5 to 0x0114b6 (0xffee bytes) 000710 1767321144.824668 trunc from 0x0114b7 to 0x03dce0 (0x2c82a bytes) 000713 1767321144.843402 mapwrite 0x027965 thru 0x029c2a (0x22c6 bytes) truncating to largest ever: 0x3fa7f 000714 1767321144.907098 trunc from 0x03dce1 to 0x03fa7e (0x1d9e bytes) 000715 1767321144.921937 trunc from 0x03fa7f to 0x01eb43 (0x20f3b bytes) 000716 1767321144.943446 trunc from 0x01eb44 to 0x0104fa (0xe649 bytes) 000718 1767321144.975935 trunc from 0x0104fb to 0x02555e (0x15064 bytes) 000721 1767321145.000996 trunc from 0x02555f to 0x0378c6 (0x12368 bytes) 000722 1767321145.036499 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x378c7): Mirror 6: 0x7f5ec8c8 (stale) Mirror 9: 0x7f5ec8c8 (stale) Mirror 11: 0x7f5ec8c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x378c7): Mirror 6: 0x7f5ec8c8 (stale) Mirror 9: 0x7f5ec8c8 (stale) Mirror 11: 0x7f5ec8c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x378c7: skipped 000724 1767321145.365460 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x378c7): Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x378c7): Mirror 6: 0x7f5ec8c8 (stale) Mirror 9: 0x7f5ec8c8 (stale) Mirror 11: 0x7f5ec8c8 (stale) CRC-32 checksum value for chunk [0, 0x378c7): Mirror 1: 0x97a1c34f Mirror 12: 0x97a1c34f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x378c7: skipped 000726 1767321146.258368 read 0x02677a thru 0x0311e8 (0xaa6f bytes) 000727 1767321146.276010 mapread 0x016cad thru 0x020ebb (0xa20f bytes) 000729 1767321146.321114 mapread 0x03480e thru 0x0378c6 (0x30b9 bytes) 000731 1767321146.353479 read 0x0231d1 thru 0x0246a9 (0x14d9 bytes) 000733 1767321146.375150 mapread 0x012f1e thru 0x013d50 (0xe33 bytes) 000734 1767321146.404431 mapwrite 0x00bd66 thru 0x013dc9 (0x8064 bytes) 000735 1767321146.703142 mapwrite 0x013d00 thru 0x0193e1 (0x56e2 bytes) 000736 1767321146.803130 mapread 0x00652e thru 0x013e4f (0xd922 bytes) 000737 1767321146.842081 mapread 0x00542a thru 0x0135d6 (0xe1ad bytes) 000738 1767321146.852198 write 0x024016 thru 0x0331f2 (0xf1dd bytes) 000739 1767321146.866599 mapread 0x023126 thru 0x02a059 (0x6f34 bytes) 000741 1767321146.876577 mapread 0x00edd5 thru 0x00f6b1 (0x8dd bytes) 000742 1767321146.878069 read 0x028cb2 thru 0x034911 (0xbc60 bytes) 000744 1767321146.897375 mapwrite 0x00c466 thru 0x014262 (0x7dfd bytes) 000745 1767321146.968651 write 0x039b28 thru 0x03ffff (0x64d8 bytes) 000746 1767321146.971468 trunc from 0x040000 to 0x01dd43 (0x222bc bytes) 000748 1767321147.006250 write 0x028dda thru 0x02b9bf (0x2be6 bytes) 000749 1767321147.007482 mapwrite 0x02b482 thru 0x02e114 (0x2c93 bytes) 000751 1767321147.111248 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e115): Mirror 9: 0x47784c4c (stale) Mirror 11: 0x47784c4c (stale) Mirror 12: 0xe524c4b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e115): Mirror 9: 0x47784c4c (stale) Mirror 11: 0x47784c4c (stale) Mirror 12: 0xe524c4b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e115: skipped 000754 1767321147.666516 write 0x022a0b thru 0x02d17f (0xa775 bytes) 000758 1767321147.714380 write 0x0104b3 thru 0x01ffa5 (0xfaf3 bytes) 000760 1767321147.766504 mapread 0x02b4fc thru 0x02bcbb (0x7c0 bytes) 000762 1767321147.776313 mapread 0x02432d thru 0x02e114 (0x9de8 bytes) 000764 1767321147.799427 write 0x039e59 thru 0x03ffff (0x61a7 bytes) 000765 1767321147.804118 mapread 0x012e8c thru 0x01c82a (0x999f bytes) 000766 1767321147.805015 write 0x00f8c4 thru 0x011b21 (0x225e bytes) 000767 1767321147.814651 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x39bd6162 (stale) Mirror 11: 0x39bd6162 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x39bd6162 (stale) Mirror 11: 0x39bd6162 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000768 1767321148.309293 mapwrite 0x02ee90 thru 0x03c6ca (0xd83b bytes) 000770 1767321148.424551 mapwrite 0x031f59 thru 0x03923d (0x72e5 bytes) 000774 1767321148.461981 mapread 0x00e239 thru 0x01aac6 (0xc88e bytes) 000775 1767321148.475915 trunc from 0x040000 to 0x02d12f (0x12ed0 bytes) 000776 1767321148.499660 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d130): Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d130): Mirror 9: 0x5e7730b6 (stale) Mirror 11: 0x5e7730b6 (stale) CRC-32 checksum value for chunk [0, 0x2d130): Mirror 1: 0x4873e57a Mirror 12: 0x4873e57a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d130: skipped 000777 1767321149.661558 read 0x00d800 thru 0x00dd2a (0x52b bytes) 000778 1767321149.671577 trunc from 0x02d130 to 0x035080 (0x7f51 bytes) 000779 1767321149.764574 write 0x00fa7b thru 0x01af11 (0xb497 bytes) 000780 1767321149.781786 read 0x00cc77 thru 0x013c6b (0x6ff5 bytes) 000781 1767321149.807360 mapwrite 0x0006c5 thru 0x00b2d7 (0xac13 bytes) 000782 1767321149.931841 write 0x039142 thru 0x03ffff (0x6ebe bytes) 000783 1767321149.937049 mapread 0x00e8ba thru 0x012934 (0x407b bytes) 000784 1767321149.941247 read 0x00a07c thru 0x0150aa (0xb02f bytes) 000785 1767321149.950296 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x39bd6162 (stale) Mirror 12: 0xcca45595 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x39bd6162 (stale) Mirror 12: 0xcca45595 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000786 1767321150.456640 trunc from 0x040000 to 0x03a5b9 (0x5a46 bytes) 000790 1767321150.482465 mapwrite 0x0172fa thru 0x01c060 (0x4d67 bytes) 000791 1767321150.544374 mapwrite 0x016a4d thru 0x0188d4 (0x1e88 bytes) 000793 1767321150.585883 read 0x039b28 thru 0x03a4cc (0x9a5 bytes) 000794 1767321150.591321 trunc from 0x03a5ba to 0x002d09 (0x378b0 bytes) 000796 1767321150.608341 trunc from 0x002d0a to 0x02131a (0x1e611 bytes) 000797 1767321150.623507 write 0x0391d0 thru 0x03ffff (0x6e30 bytes) 000798 1767321150.629672 MIRROR_RESYNC 3 mirrors 000804 1767321151.431900 write 0x0253bd thru 0x02b9d8 (0x661c bytes) 000807 1767321151.507413 mapread 0x001681 thru 0x0081c8 (0x6b48 bytes) 000810 1767321151.525039 mapread 0x03a7b0 thru 0x03ffff (0x5850 bytes) 000812 1767321151.566373 trunc from 0x040000 to 0x026e9f (0x19160 bytes) 000814 1767321151.606465 mapread 0x00e149 thru 0x00e776 (0x62e bytes) 000815 1767321151.609831 write 0x02f325 thru 0x034500 (0x51dc bytes) 000818 1767321151.612151 mapread 0x007cd4 thru 0x009d08 (0x2035 bytes) 000820 1767321151.615627 mapwrite 0x0200d9 thru 0x02f33a (0xf262 bytes) 000821 1767321151.690548 mapwrite 0x01f526 thru 0x022517 (0x2ff2 bytes) 000822 1767321151.728132 mapwrite 0x001dfb thru 0x0056fb (0x3901 bytes) 000824 1767321151.767554 read 0x023a3f thru 0x02b47b (0x7a3d bytes) 000825 1767321151.768950 trunc from 0x034501 to 0x00e149 (0x263b7 bytes) 000826 1767321151.788774 write 0x018823 thru 0x0271bd (0xe99b bytes) 000828 1767321151.794964 write 0x030d99 thru 0x037b7c (0x6de4 bytes) 000829 1767321151.802398 MIRROR_RESYNC 3 mirrors 000830 1767321152.508166 MIRROR_RESYNC 3 mirrors 000832 1767321153.044260 read 0x02ffd4 thru 0x037b7c (0x7ba9 bytes) 000834 1767321153.059920 trunc from 0x037b7d to 0x0351d4 (0x29a8 bytes) 000835 1767321153.161319 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x351d5): Mirror 12: 0xe48bcf92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351d5): Mirror 12: 0xe48bcf92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351d5: skipped 000836 1767321153.776138 mapread 0x032d71 thru 0x033ed8 (0x1168 bytes) 000839 1767321153.790349 trunc from 0x0351d5 to 0x0060c2 (0x2f112 bytes) 000840 1767321153.813309 mapwrite 0x03e7a4 thru 0x03ffff (0x185c bytes) 000841 1767321153.944674 mapwrite 0x02505d thru 0x025916 (0x8ba bytes) 000842 1767321153.999152 read 0x0197a1 thru 0x020d13 (0x7573 bytes) 000843 1767321154.008633 read 0x021475 thru 0x02d9d4 (0xc560 bytes) 000844 1767321154.009482 read 0x01059f thru 0x017966 (0x73c8 bytes) 000845 1767321154.024522 trunc from 0x040000 to 0x03d654 (0x29ab bytes) 000847 1767321154.050454 write 0x01c69c thru 0x0205d3 (0x3f38 bytes) 000848 1767321154.057312 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d655): Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d655): Mirror 12: 0xc4259ac5 (stale) CRC-32 checksum value for chunk [0, 0x3d655): Mirror 1: 0x1f963f0f Mirror 13: 0x1f963f0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d655: skipped 000849 1767321155.659202 trunc from 0x03d655 to 0x00e3d9 (0x2f27b bytes) 000853 1767321155.960307 mapwrite 0x02672a thru 0x02d9ff (0x72d6 bytes) 000857 1767321156.142095 write 0x00226c thru 0x011cb5 (0xfa4a bytes) 000859 1767321156.175666 mapread 0x01ac85 thru 0x01b61e (0x99a bytes) 000860 1767321156.189672 write 0x00a6d6 thru 0x00e996 (0x42c1 bytes) 000861 1767321156.190865 mapwrite 0x00639f thru 0x00962f (0x3291 bytes) 000863 1767321156.278043 trunc from 0x02da00 to 0x001dee (0x2bc11 bytes) 000865 1767321156.303450 trunc from 0x001def to 0x01bac1 (0x19cd3 bytes) 000867 1767321156.361620 trunc from 0x01bac2 to 0x0342ee (0x1882d bytes) 000868 1767321156.398194 write 0x00c881 thru 0x00f6b2 (0x2e32 bytes) 000869 1767321156.439180 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x342ef): Mirror 13: 0xde4c8f57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342ef): Mirror 13: 0xde4c8f57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342ef: skipped 000870 1767321156.831399 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000873 1767321157.030131 read 0x02fb02 thru 0x0342ee (0x47ed bytes) 000879 1767321157.064296 write 0x03dde5 thru 0x03ffff (0x221b bytes) 000880 1767321157.074346 write 0x00e9d4 thru 0x0155f5 (0x6c22 bytes) 000881 1767321157.104237 mapread 0x037112 thru 0x03f697 (0x8586 bytes) 000883 1767321157.127750 mapread 0x00c35b thru 0x013b69 (0x780f bytes) 000886 1767321157.149933 MIRROR_EXTEND to 2 mirrors 000887 1767321158.146773 trunc from 0x040000 to 0x027ec5 (0x1813a bytes) 000888 1767321158.300445 mapread 0x016c3a thru 0x020892 (0x9c59 bytes) 000890 1767321158.324160 trunc from 0x027ec6 to 0x015685 (0x12840 bytes) 000893 1767321158.350969 mapwrite 0x00e8c0 thru 0x01b481 (0xcbc2 bytes) 000894 1767321158.465547 mapwrite 0x02caad thru 0x034e44 (0x8398 bytes) 000896 1767321158.573445 write 0x02cd53 thru 0x02ea3c (0x1cea bytes) 000897 1767321158.575889 write 0x01a3a2 thru 0x01a4c7 (0x126 bytes) 000898 1767321158.578030 mapwrite 0x0245fa thru 0x025cbc (0x16c3 bytes) 000900 1767321158.668831 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000907 1767321158.832004 read 0x0204c1 thru 0x025122 (0x4c62 bytes) 000910 1767321158.841326 trunc from 0x034e45 to 0x038584 (0x3740 bytes) 000913 1767321158.861450 mapread 0x019009 thru 0x01e4eb (0x54e3 bytes) 000914 1767321158.881196 read 0x0264b7 thru 0x026a36 (0x580 bytes) 000916 1767321158.896431 trunc from 0x038585 to 0x0396cf (0x114b bytes) 000917 1767321158.925015 MIRROR_EXTEND to 2 mirrors 000918 1767321159.589614 read 0x033caa thru 0x039193 (0x54ea bytes) 000921 1767321159.605131 write 0x02bfa5 thru 0x039e8a (0xdee6 bytes) 000922 1767321159.723369 trunc from 0x039e8b to 0x0316a0 (0x87ea bytes) 000923 1767321159.752071 write 0x01f4a4 thru 0x02406c (0x4bc9 bytes) 000924 1767321159.765804 mapwrite 0x035716 thru 0x0398a3 (0x418e bytes) 000925 1767321159.891171 mapwrite 0x021bbd thru 0x0264eb (0x492f bytes) 000928 1767321159.958599 mapread 0x0007cb thru 0x00e5aa (0xdde0 bytes) 000929 1767321159.972940 mapwrite 0x01c9a9 thru 0x01e431 (0x1a89 bytes) 000930 1767321160.036529 write 0x02d82a thru 0x02ec46 (0x141d bytes) 000931 1767321160.037479 trunc from 0x0398a4 to 0x00b48e (0x2e415 bytes) 000932 1767321160.082637 read 0x008dc7 thru 0x00b48e (0x26c8 bytes) 000935 1767321160.087250 trunc from 0x00b48f to 0x01bc43 (0x107b5 bytes) 000937 1767321160.106920 mapread 0x001e98 thru 0x00ca49 (0xabb2 bytes) 000940 1767321160.131004 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) 000943 1767321160.296047 trunc from 0x01bc44 to 0x003305 (0x1893e bytes) 000944 1767321160.312260 write 0x0311a6 thru 0x039010 (0x7e6b bytes) 000945 1767321160.313594 mapread 0x036c5f thru 0x039010 (0x23b2 bytes) 000946 1767321160.320425 MIRROR_EXTEND to 2 mirrors 000947 1767321160.927291 mapread 0x010b0e thru 0x0153d5 (0x48c8 bytes) 000948 1767321160.950604 mapread 0x0076ac thru 0x00c171 (0x4ac6 bytes) 000951 1767321160.963210 trunc from 0x039011 to 0x03e272 (0x5262 bytes) 000953 1767321161.039003 write 0x005b95 thru 0x00f905 (0x9d71 bytes) 000956 1767321161.049001 read 0x01b507 thru 0x01bd54 (0x84e bytes) 000957 1767321161.058972 mapwrite 0x0035fc thru 0x004758 (0x115d bytes) 000958 1767321161.143877 mapread 0x0299f6 thru 0x02a6cb (0xcd6 bytes) 000959 1767321161.154460 mapread 0x004ac9 thru 0x00ba9f (0x6fd7 bytes) 000960 1767321161.155444 mapwrite 0x03d7b8 thru 0x03ffff (0x2848 bytes) 000961 1767321161.230312 write 0x01117b thru 0x017ece (0x6d54 bytes) 000962 1767321161.240420 trunc from 0x040000 to 0x016a4d (0x295b2 bytes) 000963 1767321161.271410 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x16a4e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16a4e): Mirror 2: 0x3076e1a1 (stale) CRC-32 checksum value for chunk [0, 0x16a4e): Mirror 1: 0x4687b0f9 Mirror 3: 0x4687b0f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16a4e: skipped 000964 1767321162.330611 MIRROR_RESYNC 3 mirrors 000965 1767321163.223284 MIRROR_EXTEND to 4 mirrors 000967 1767321164.439791 trunc from 0x016a4e to 0x014ab3 (0x1f9a bytes) 000970 1767321164.691729 write 0x00e86a thru 0x016d80 (0x8517 bytes) 000972 1767321164.701350 read 0x016166 thru 0x016d80 (0xc1b bytes) 000973 1767321164.701944 write 0x02a270 thru 0x02bdb5 (0x1b46 bytes) 000974 1767321164.702876 mapwrite 0x012452 thru 0x017804 (0x53b3 bytes) 000975 1767321164.817409 mapwrite 0x038d87 thru 0x03de16 (0x5090 bytes) 000976 1767321164.977830 mapwrite 0x0248a0 thru 0x0268f1 (0x2052 bytes) 000978 1767321165.141905 mapread 0x007f1f thru 0x00f59f (0x7681 bytes) 000982 1767321165.164890 trunc from 0x03de17 to 0x007f3e (0x35ed8 bytes) 000984 1767321165.230948 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x7f3f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7f3f): Mirror 2: 0x457e4505 (stale) Mirror 3: 0x457e4505 (stale) Mirror 4: 0x457e4505 (stale) CRC-32 checksum value for chunk [0, 0x7f3f): Mirror 1: 0x457e4505 Mirror 5: 0x457e4505 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7f3f: skipped 000985 1767321166.330560 mapread 0x006265 thru 0x007f3e (0x1cda bytes) 000986 1767321166.409420 read 0x007f1d thru 0x007f3e (0x22 bytes) 000987 1767321166.419451 mapread 0x0039fb thru 0x007f3e (0x4544 bytes) 000988 1767321166.462214 trunc from 0x007f3f to 0x0034be (0x4a80 bytes) 000989 1767321166.648969 trunc from 0x0034bf to 0x03d028 (0x39b6a bytes) 000990 1767321166.703564 write 0x02960f thru 0x02c20c (0x2bfe bytes) 000991 1767321166.719375 write 0x03f563 thru 0x03ffff (0xa9d bytes) 000993 1767321166.721491 write 0x011b15 thru 0x02040a (0xe8f6 bytes) 000994 1767321166.742732 trunc from 0x040000 to 0x01e184 (0x21e7b bytes) 000995 1767321166.771121 trunc from 0x01e185 to 0x03dccf (0x1fb4b bytes) 000997 1767321166.816648 read 0x0378e8 thru 0x03a87e (0x2f97 bytes) 000999 1767321166.839666 mapread 0x016f8c thru 0x02020d (0x9282 bytes) 001000 1767321166.849647 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dcd0): Mirror 3: 0x453b22e2 (stale) Mirror 4: 0x453b22e2 (stale) Mirror 5: 0x512f4286 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dcd0): Mirror 3: 0x453b22e2 (stale) Mirror 4: 0x453b22e2 (stale) Mirror 5: 0x512f4286 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dcd0: skipped All operations completed A-OK! PASS 70a (95s) == sanity-flr test 71: check mirror extend parent fid ==== 21:32:54 (1767321174) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 21:32:57 (1767321177) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.331843 s, 6.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286831 s, 3.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x240:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x240:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x23d:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 21:33:06 (1767321186) Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2885184 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 2458304 .. Extending file size to 4527264 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5981152 .. Extending file size to 5763904 .. Extending file size to 7786368 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7836320 .. Extending file size to 8361440 .. Extending file size to 8227232 .. Extending file size to 8822688 .. Extending file size to 8997888 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9154048 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57871 57873 57874 57875 57877 done 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) 192.168.202.120@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:34:25 (1767321265) Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6987808 .. Extending file size to 7223712 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7876608 .. Extending file size to 8023584 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8182528 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8816480 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8996064 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8742976 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9378336 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59689 59690 59691 59692 59693 done 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) 192.168.202.120@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:35:47 (1767321347) Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197186 s, 10.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.420415 s, 5.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3714161a (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:36:09 (1767321369) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124146 s, 16.9 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 21:36:18 (1767321378) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x256:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x256:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x257:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x252:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:36:27 (1767321387) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.358477 s, 5.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x258:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x253:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x254:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:36:36 (1767321396) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.472112 s, 4.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x259:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x255:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x259:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x255:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x256:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:36:46 (1767321406) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.334219 s, 6.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x257:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x257:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x258:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:37:21 (1767321441) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg220-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145487 s, 7.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x259:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x259:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25d:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:38:17 (1767321497) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg220-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.352538 s, 5.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:39:11 (1767321551) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg220-server mds-ost sync done. Creating new pool test_204f oleg220-server: Pool lustre.test_204f created Adding targets to pool oleg220-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg220-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.340594 s, 6.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x25f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg220-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg220-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg220-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:40:22 (1767321622) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:40:30 (1767321630) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:40:36 (1767321636) oleg220-server: Pool lustre.test_206 created Adding targets to pool oleg220-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg220-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg220-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg220-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg220-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:40:59 (1767321659) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10989 s, 9.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473673 s, 22.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd47019d9 Mirror 3: 0xd47019d9 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:41:11 (1767321671) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:41:15 (1767321675) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:41:20 (1767321680) Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.170537 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xa00000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xa00000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xa00000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2370000 Read this (4 bytes): BBBB 192.168.202.120@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg220-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:41:59 (1767321719) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0400325 s, 205 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:42:06 (1767321726) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388153 s, 27.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fc lmm_fid: [0x200000404:0x1fc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fc lmm_fid: [0x200000404:0x1fc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x269:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834428 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-01-01 21:42:08.000000000 -0500 Modify: 2026-01-01 21:42:08.000000000 -0500 Change: 2026-01-01 21:42:09.000000000 -0500 Birth: 2026-01-01 21:42:08.000000000 -0500 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:42:14 (1767321734) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:42:18 (1767321738) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.670139 s, 61.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0984687 s, 41.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Jan 1 21:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fe:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jan 1 21:42 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:42:26 (1767321746) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:42:32 (1767321752) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1675 sec =========== 21:42:41 (1767321761) === sanity-flr: start cleanup 21:42:42 (1767321762) === === sanity-flr: finish cleanup 21:42:47 (1767321767) ===