-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 16:37:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg138-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:37:26 (1787863046) === oleg138-client.virtnet: executing check_config_client /mnt/lustre oleg138-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg138-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a6c450eb800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a6c450eb800.idle_timeout=debug disable quota as required oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:38:05 (1787863085) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7622] == sanity-flr test 0a: lfs mirror create with -N option == 16:38:08 (1787863088) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:38:27 (1787863107) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:38:31 (1787863111) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:38:34 (1787863114) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:38:59 (1787863139) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:39:03 (1787863143) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 16:39:07 (1787863147) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8a6c450eb800.stats=clear osc.lustre-OST0001-osc-ffff8a6c450eb800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:39:15 (1787863155) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:39:27 (1787863167) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:39:35 (1787863175) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:39:39 (1787863179) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:39:48 (1787863188) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 16:40:00 (1787863200) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:40:08 (1787863208) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:40:13 (1787863213) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:40:21 (1787863221) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:40:31 (1787863231) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:40:39 (1787863239) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.09738 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11929 s, 8.8 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:40:51 (1787863251) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.202949 s, 25.8 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:41:02 (1787863262) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:41:13 (1787863273) Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a6c450eb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a6c450eb800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a6c450eb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a6c450eb800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (82s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:42:35 (1787863355) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.421383 s, 14.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a6c450eb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a6c450eb800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:43:19 (1787863399) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a6c450eb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a6c450eb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a6c450eb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a6c450eb800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (90s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:44:49 (1787863489) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.65731 s, 9.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 16:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.215s user 0m0.002s sys 0m0.217s Stopping /mnt/lustre-ost1 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a6c450eb800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a6c450eb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.087s user 0m0.002s sys 0m0.180s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg138-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg138-server oleg138-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a6c450eb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a6c450eb800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.064s user 0m0.000s sys 0m0.145s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg138-server: oleg138-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg138-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a6c450eb800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a6c450eb800.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:46:35 (1787863595) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:46:38 (1787863598) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:46:42 (1787863602) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 16:46:45 (1787863605) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 16:46:55 (1787863615) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.684387 s, 18.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:47:10 (1787863630) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260027 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311914 s, 33.6 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:47:19 (1787863639) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:47:28 (1787863648) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122916 s, 833 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113257 s, 904 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00384486 s, 266 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00230278 s, 445 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 16:47:39 (1787863659) create test files with size 15744897 .. ff3b79cc18f05923536426beb70209d0 - 22b1c380edc6a1a53a91a4bac69dc36a - 56b5efbf2a76ab38f5e53a6d294f88bb - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 16:48:08 (1787863688) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:48:27 (1787863707) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:48:34 (1787863714) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101671 s, 10.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046498 s, 22.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc3:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 16:48:44 (1787863724) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20824 s, 10.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.416858 s, 10.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142004 s, 14.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 16:49:02 (1787863742) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 16:49:05 (1787863745) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:49:09 (1787863749) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202037 s, 51.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 16:49:17 (1787863757) llite.lustre-ffff8a6c43d6a800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 16:49:24 (1787863764) llite.lustre-ffff8a6c43d6a800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379526 s, 27.6 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464253 s, 22.6 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 16:49:33 (1787863773) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:49:37 (1787863777) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:49:50 (1787863790) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.316597 s, 33.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:50:02 (1787863802) llite.lustre-ffff8a6c47e34800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.307508 s, 34.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg138-client.virtnet /mnt/lustre (opts:) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8a6c60686800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:50:19 (1787863819) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (48s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:51:07 (1787863867) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:51:44 (1787863904) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 16:51:48 (1787863908) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.179098 s, 17.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.294126 s, 10.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x4823a302 (nosync,stale) Mirror 2: 0x4823a302 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 84508a6593e8fb6d816e76c9d666d56e - mirror 1 checksum: b77da82aa0bfb9c630f72c773a5447ad - mirror 2 checksum: 84508a6593e8fb6d816e76c9d666d56e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T16:51:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T16:51:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 84508a6593e8fb6d816e76c9d666d56e - mirror 1 checksum: b77da82aa0bfb9c630f72c773a5447ad - mirror 2 checksum: b77da82aa0bfb9c630f72c773a5447ad - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T16:51:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T16:51:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe4:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:52:02 (1787863922) Failing mds1 on oleg138-server Stopping /mnt/lustre-mds1 (opts:) on oleg138-server 16:52:14 (1787863934) shut down facet: mds1 facet_host: oleg138-server facet_failover_host: oleg138-server Failover mds1 to oleg138-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 16:52:48 (1787863968) targets are mounted 16:52:48 (1787863968) facet_failover done oleg138-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341858 s, 30.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe5:0x0] } PASS 50A (65s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:53:07 (1787863987) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:53:10 (1787863990) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:53:14 (1787863994) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0413077 s, 397 kB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:53:24 (1787864004) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:53:29 (1787864009) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.027958 s, 147 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.161782 s, 25.3 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:53:40 (1787864020) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:54:04 (1787864044) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:54:34 (1787864074) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:54:59 (1787864099) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:55:23 (1787864123) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 16:55:31 (1787864131) mirror create pid 55589 mirror split pid 55590 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55589 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55590 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 16:56:39 (1787864199) Seed set to 4201 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1787864201.399862 mapwrite 0x03213d thru 0x03d0ac (0xaf70 bytes) 000005 1787864201.572764 read 0x023098 thru 0x02bbba (0x8b23 bytes) 000006 1787864201.606951 mapwrite 0x031e25 thru 0x039d9c (0x7f78 bytes) 000007 1787864201.733594 read 0x02fb41 thru 0x03b328 (0xb7e8 bytes) 000009 1787864201.735231 write 0x01a4ca thru 0x01a9f4 (0x52b bytes) truncating to largest ever: 0x60 000014 1787864201.768952 trunc from 0x03d0ad to 0x00005f (0x3d04d bytes) 000018 1787864201.801583 mapread 0x00005e thru 0x00005f (0x2 bytes) 000020 1787864201.805876 mapread 0x00003b thru 0x00005f (0x25 bytes) 000021 1787864201.806783 read 0x000046 thru 0x00005f (0x1a bytes) 000023 1787864201.810784 mapread 0x000047 thru 0x00005f (0x19 bytes) 000024 1787864201.811748 mapwrite 0x01fce1 thru 0x027586 (0x78a6 bytes) 000025 1787864201.946751 mapread 0x022279 thru 0x027586 (0x530e bytes) 000026 1787864201.950860 write 0x02bb46 thru 0x037002 (0xb4bd bytes) 000028 1787864201.959129 write 0x017f24 thru 0x01c280 (0x435d bytes) truncating to largest ever: 0x30630 000029 1787864201.963571 trunc from 0x037003 to 0x03062f (0x69d3 bytes) 000032 1787864201.994387 write 0x00bbd0 thru 0x0117d0 (0x5c01 bytes) 000033 1787864201.996683 mapwrite 0x00ab9a thru 0x00d7c7 (0x2c2e bytes) 000035 1787864202.068512 mapwrite 0x02a83e thru 0x02b8b2 (0x1075 bytes) 000036 1787864202.175767 read 0x000c8d thru 0x009d70 (0x90e4 bytes) 000041 1787864202.197460 mapread 0x028a68 thru 0x03062f (0x7bc8 bytes) 000042 1787864202.200212 read 0x021555 thru 0x03062f (0xf0db bytes) 000045 1787864202.209196 read 0x00d639 thru 0x01c912 (0xf2da bytes) 000046 1787864202.211486 mapread 0x021f71 thru 0x02bf58 (0x9fe8 bytes) 000048 1787864202.213576 read 0x00b81c thru 0x019906 (0xe0eb bytes) 000049 1787864202.224976 mapread 0x025828 thru 0x02c5fb (0x6dd4 bytes) 000050 1787864202.227106 write 0x03d0c0 thru 0x03ffff (0x2f40 bytes) 000051 1787864202.229068 mapread 0x007369 thru 0x013246 (0xbede bytes) 000052 1787864202.231236 write 0x027989 thru 0x034db8 (0xd430 bytes) 000054 1787864202.247075 trunc from 0x040000 to 0x010b97 (0x2f468 bytes) 000055 1787864202.288992 mapwrite 0x0028d8 thru 0x00d271 (0xa99a bytes) 000056 1787864202.398250 read 0x00684a thru 0x00cea1 (0x6658 bytes) 000059 1787864202.405774 read 0x008339 thru 0x010b97 (0x885f bytes) 000060 1787864202.407778 write 0x029ea2 thru 0x033ce7 (0x9e46 bytes) 000061 1787864202.410436 mapread 0x01c236 thru 0x022ad3 (0x689e bytes) truncating to largest ever: 0x37bdb 000063 1787864202.424390 trunc from 0x033ce8 to 0x037bda (0x3ef3 bytes) 000064 1787864202.446386 read 0x022436 thru 0x025c3e (0x3809 bytes) 000067 1787864202.449394 write 0x023336 thru 0x02e194 (0xae5f bytes) 000069 1787864202.452879 write 0x01ecc8 thru 0x0292c6 (0xa5ff bytes) 000072 1787864202.471545 mapread 0x00ce46 thru 0x016538 (0x96f3 bytes) truncating to largest ever: 0x37d22 000074 1787864202.476966 trunc from 0x037bdb to 0x037d21 (0x147 bytes) 000075 1787864202.509916 read 0x022482 thru 0x0286fe (0x627d bytes) 000079 1787864202.511994 read 0x00e8b7 thru 0x01ce4a (0xe594 bytes) 000080 1787864202.518044 write 0x02dcbf thru 0x036f4c (0x928e bytes) 000082 1787864202.528802 mapwrite 0x0175a9 thru 0x023683 (0xc0db bytes) 000083 1787864202.625129 write 0x017530 thru 0x024f42 (0xda13 bytes) 000085 1787864202.628374 mapwrite 0x03f78b thru 0x03fcef (0x565 bytes) 000087 1787864202.740486 mapread 0x02ea9d thru 0x03b6d8 (0xcc3c bytes) 000088 1787864202.741725 mapread 0x0395b5 thru 0x03fcef (0x673b bytes) 000090 1787864202.744433 read 0x030161 thru 0x033eca (0x3d6a bytes) 000091 1787864202.745508 read 0x031567 thru 0x03e173 (0xcc0d bytes) 000092 1787864202.746832 trunc from 0x03fcf0 to 0x00a536 (0x357b9 bytes) 000093 1787864202.782395 write 0x01e8aa thru 0x02e6a4 (0xfdfb bytes) 000094 1787864202.788139 write 0x03ab35 thru 0x03ffff (0x54cb bytes) 000095 1787864202.791528 mapwrite 0x021d13 thru 0x026cbb (0x4fa9 bytes) 000096 1787864202.900424 read 0x010bb3 thru 0x0121c4 (0x1612 bytes) 000097 1787864202.941407 mapwrite 0x01efee thru 0x0292fc (0xa30f bytes) 000099 1787864202.997209 mapwrite 0x01ba14 thru 0x025090 (0x967d bytes) 000103 1787864203.064732 write 0x02ed45 thru 0x03a355 (0xb611 bytes) 000105 1787864203.068585 read 0x02d734 thru 0x039e5c (0xc729 bytes) 000107 1787864203.070288 mapwrite 0x03aaf3 thru 0x03ffff (0x550d bytes) 000110 1787864203.151517 mapread 0x032278 thru 0x03693c (0x46c5 bytes) 000111 1787864203.152942 trunc from 0x040000 to 0x00c683 (0x3397c bytes) 000113 1787864203.204849 MIRROR_EXTEND to 2 mirrors 000115 1787864204.192408 write 0x00928a thru 0x010e09 (0x7b80 bytes) 000121 1787864204.268397 mapwrite 0x0223d3 thru 0x030992 (0xe5c0 bytes) 000124 1787864204.374001 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) 000125 1787864204.547173 mapwrite 0x03dc9f thru 0x03ffff (0x2361 bytes) 000126 1787864204.638587 read 0x014e1d thru 0x01f75f (0xa943 bytes) 000127 1787864204.653594 mapwrite 0x01463a thru 0x0176d1 (0x3098 bytes) 000128 1787864204.705758 write 0x0354f9 thru 0x037cf6 (0x27fe bytes) 000131 1787864204.721043 mapwrite 0x00c5a3 thru 0x00d384 (0xde2 bytes) 000133 1787864204.786113 write 0x001ced thru 0x0059fb (0x3d0f bytes) 000136 1787864204.795806 mapwrite 0x02183d thru 0x023f32 (0x26f6 bytes) 000138 1787864204.859476 write 0x028441 thru 0x02a246 (0x1e06 bytes) 000141 1787864204.869553 trunc from 0x040000 to 0x017ba4 (0x2845b bytes) 000143 1787864204.888468 mapwrite 0x004237 thru 0x00e676 (0xa440 bytes) 000144 1787864204.937173 write 0x017a57 thru 0x01a4a2 (0x2a4c bytes) 000146 1787864204.940211 trunc from 0x01a4a3 to 0x0217fa (0x7358 bytes) 000149 1787864204.958113 write 0x017b31 thru 0x026a2d (0xeefd bytes) 000151 1787864204.966451 mapwrite 0x039854 thru 0x03ffff (0x67ac bytes) 000152 1787864205.041541 mapread 0x01f47f thru 0x020da3 (0x1925 bytes) 000155 1787864205.051279 mapread 0x01740b thru 0x0273db (0xffd1 bytes) 000156 1787864205.067306 mapwrite 0x00c875 thru 0x00d336 (0xac2 bytes) 000158 1787864205.101308 mapread 0x0293e0 thru 0x030f9b (0x7bbc bytes) 000159 1787864205.107185 read 0x03f613 thru 0x03ffff (0x9ed bytes) 000160 1787864205.107960 read 0x02bee4 thru 0x0380a5 (0xc1c2 bytes) 000161 1787864205.108445 mapread 0x0159c2 thru 0x01e5be (0x8bfd bytes) 000164 1787864205.110162 mapread 0x01ae3b thru 0x0293d7 (0xe59d bytes) 000165 1787864205.111995 mapwrite 0x023ec7 thru 0x032ae4 (0xec1e bytes) 000168 1787864205.180291 trunc from 0x040000 to 0x021bc8 (0x1e437 bytes) 000170 1787864205.208241 mapwrite 0x02f612 thru 0x033d14 (0x4703 bytes) 000172 1787864205.309339 MIRROR_EXTEND to 2 mirrors 000174 1787864205.959503 trunc from 0x033d15 to 0x031ea2 (0x1e72 bytes) 000175 1787864206.088052 write 0x02b279 thru 0x037a73 (0xc7fb bytes) 000176 1787864206.096659 read 0x01ea99 thru 0x02939a (0xa902 bytes) 000177 1787864206.108948 write 0x02eeec thru 0x037d7c (0x8e91 bytes) 000178 1787864206.110499 mapwrite 0x0151da thru 0x01b542 (0x6369 bytes) 000180 1787864206.189409 MIRROR_RESYNC 2 mirrors 000181 1787864207.492045 mapwrite 0x039c5e thru 0x03ffff (0x63a2 bytes) 000182 1787864207.742446 mapread 0x03db62 thru 0x03ffff (0x249e bytes) 000184 1787864207.744542 mapread 0x02ba59 thru 0x03874d (0xccf5 bytes) 000186 1787864207.768070 mapread 0x03f520 thru 0x03ffff (0xae0 bytes) 000188 1787864207.776797 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) 000190 1787864207.965215 read 0x014544 thru 0x01879f (0x425c bytes) 000191 1787864207.977851 trunc from 0x040000 to 0x0103d7 (0x2fc28 bytes) 000192 1787864207.994139 read 0x00d603 thru 0x0103d7 (0x2dd5 bytes) 000195 1787864208.011682 read 0x0086ec thru 0x00a7a3 (0x20b8 bytes) 000199 1787864208.016051 read 0x0030e8 thru 0x00d31a (0xa233 bytes) 000201 1787864208.017276 read 0x00e188 thru 0x0103d7 (0x2250 bytes) 000202 1787864208.018528 write 0x035baf thru 0x0378d3 (0x1d25 bytes) 000203 1787864208.026058 trunc from 0x0378d4 to 0x01aca3 (0x1cc30 bytes) 000204 1787864208.061265 mapwrite 0x0191fd thru 0x028b56 (0xf95a bytes) 000208 1787864208.143328 read 0x00c017 thru 0x010797 (0x4781 bytes) 000211 1787864208.146112 write 0x01028d thru 0x015896 (0x560a bytes) 000218 1787864208.154807 read 0x01602a thru 0x022247 (0xc21e bytes) 000220 1787864208.163051 MIRROR_EXTEND to 2 mirrors 000222 1787864209.329474 MIRROR_EXTEND to 3 mirrors 000223 1787864210.274453 trunc from 0x028b57 to 0x022588 (0x65ce bytes) 000224 1787864210.546492 mapread 0x00e1e1 thru 0x017c97 (0x9ab7 bytes) 000225 1787864210.595301 read 0x00ff98 thru 0x01b2cc (0xb335 bytes) 000227 1787864210.603580 mapwrite 0x01f145 thru 0x025634 (0x64f0 bytes) 000228 1787864210.711047 write 0x027e58 thru 0x036dc5 (0xef6e bytes) 000230 1787864210.714817 write 0x0318c0 thru 0x033d41 (0x2482 bytes) 000231 1787864210.716230 read 0x01eb1f thru 0x02800c (0x94ee bytes) 000232 1787864210.736841 mapread 0x0320bb thru 0x036dc5 (0x4d0b bytes) 000233 1787864210.743010 read 0x022c3b thru 0x030d6c (0xe132 bytes) 000234 1787864210.748308 mapread 0x0203db thru 0x027e98 (0x7abe bytes) 000236 1787864210.782218 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36dc6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36dc6): Mirror 2: 0xaa899887 (stale) Mirror 3: 0xaa899887 (stale) CRC-32 checksum value for chunk [0, 0x36dc6): Mirror 1: 0x5bdbac35 Mirror 4: 0x5bdbac35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36dc6: skipped 000237 1787864212.447024 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36dc6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36dc6): Mirror 2: 0xaa899887 (stale) Mirror 3: 0xaa899887 (stale) CRC-32 checksum value for chunk [0, 0x36dc6): Mirror 1: 0x5bdbac35 Mirror 4: 0x5bdbac35 Mirror 5: 0x5bdbac35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36dc6: skipped 000238 1787864214.213760 read 0x02ed7b thru 0x033b91 (0x4e17 bytes) 000239 1787864214.247167 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36dc6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36dc6): Mirror 2: 0xaa899887 (stale) Mirror 3: 0xaa899887 (stale) CRC-32 checksum value for chunk [0, 0x36dc6): Mirror 1: 0x5bdbac35 Mirror 4: 0x5bdbac35 Mirror 5: 0x5bdbac35 Mirror 6: 0x5bdbac35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36dc6: skipped 000240 1787864215.580259 mapwrite 0x00fd9c thru 0x011a88 (0x1ced bytes) 000242 1787864215.734223 trunc from 0x036dc6 to 0x0116f1 (0x256d4 bytes) 000244 1787864215.756187 trunc from 0x0116f2 to 0x01f5df (0xdeee bytes) 000246 1787864215.773301 read 0x00cc1d thru 0x00ce42 (0x226 bytes) 000247 1787864215.774270 mapwrite 0x03cc77 thru 0x03ffff (0x3389 bytes) 000248 1787864215.887380 mapwrite 0x03a49c thru 0x03ffff (0x5b64 bytes) 000250 1787864216.065323 trunc from 0x040000 to 0x01aad2 (0x2552d bytes) 000251 1787864216.089101 write 0x007769 thru 0x015f21 (0xe7b9 bytes) 000253 1787864216.111335 mapread 0x0099eb thru 0x00b7ac (0x1dc2 bytes) 000254 1787864216.113230 read 0x00bcbd thru 0x00f2d9 (0x361d bytes) 000255 1787864216.115216 write 0x0009dc thru 0x005ef2 (0x5517 bytes) 000256 1787864216.158235 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1aad3): Mirror 3: 0xa926164e (stale) Mirror 4: 0xa926164e (stale) Mirror 5: 0xa926164e (stale) Mirror 6: 0xa926164e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1aad3): Mirror 3: 0xa926164e (stale) Mirror 4: 0xa926164e (stale) Mirror 5: 0xa926164e (stale) Mirror 6: 0xa926164e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1aad3: skipped 000257 1787864216.625930 mapread 0x018457 thru 0x01aad2 (0x267c bytes) 000258 1787864216.688789 read 0x0121a8 thru 0x0129f9 (0x852 bytes) 000259 1787864216.708787 read 0x015865 thru 0x01aad2 (0x526e bytes) 000261 1787864216.726319 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1aad3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1aad3): Mirror 3: 0xa926164e (stale) Mirror 4: 0xa926164e (stale) Mirror 5: 0xa926164e (stale) Mirror 6: 0xa926164e (stale) CRC-32 checksum value for chunk [0, 0x1aad3): Mirror 1: 0x827e1553 Mirror 7: 0x827e1553 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1aad3: skipped 000262 1787864218.337858 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1aad3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1aad3): Mirror 3: 0xa926164e (stale) Mirror 4: 0xa926164e (stale) Mirror 5: 0xa926164e (stale) Mirror 6: 0xa926164e (stale) CRC-32 checksum value for chunk [0, 0x1aad3): Mirror 1: 0x827e1553 Mirror 7: 0x827e1553 Mirror 8: 0x827e1553 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1aad3: skipped 000263 1787864219.913821 mapread 0x0002bf thru 0x0002e4 (0x26 bytes) 000264 1787864219.921579 read 0x005467 thru 0x011794 (0xc32e bytes) 000266 1787864219.938574 read 0x00512c thru 0x007261 (0x2136 bytes) 000267 1787864219.938965 mapread 0x00e5e6 thru 0x014013 (0x5a2e bytes) 000270 1787864219.960111 mapread 0x00d8c5 thru 0x012e44 (0x5580 bytes) 000272 1787864219.971991 MIRROR_RESYNC 7 mirrors 000273 1787864221.517867 mapread 0x001397 thru 0x00c52e (0xb198 bytes) 000274 1787864221.541442 mapread 0x013710 thru 0x01aad2 (0x73c3 bytes) 000276 1787864221.553039 read 0x00a281 thru 0x0111c6 (0x6f46 bytes) 000278 1787864221.553844 mapwrite 0x006564 thru 0x00b9de (0x547b bytes) 000279 1787864221.731497 write 0x034328 thru 0x03ffff (0xbcd8 bytes) 000280 1787864221.734181 read 0x007ede thru 0x00f31b (0x743e bytes) 000285 1787864221.737548 mapread 0x01d88b thru 0x02b7e1 (0xdf57 bytes) 000286 1787864221.751009 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x61799a28 (stale) Mirror 5: 0x61799a28 (stale) Mirror 6: 0x61799a28 (stale) Mirror 7: 0x61799a28 (stale) Mirror 8: 0x61799a28 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x61799a28 (stale) Mirror 5: 0x61799a28 (stale) Mirror 6: 0x61799a28 (stale) Mirror 7: 0x61799a28 (stale) Mirror 8: 0x61799a28 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000287 1787864222.207089 MIRROR_RESYNC 6 mirrors 000289 1787864224.305147 trunc from 0x040000 to 0x026300 (0x19cff bytes) 000292 1787864224.471994 trunc from 0x026301 to 0x01cf33 (0x93cd bytes) 000293 1787864224.510840 trunc from 0x01cf34 to 0x01c15d (0xdd6 bytes) 000295 1787864224.538886 write 0x033281 thru 0x03c365 (0x90e5 bytes) 000296 1787864224.542201 trunc from 0x03c366 to 0x00b4dc (0x30e89 bytes) 000297 1787864224.567643 write 0x01cbc4 thru 0x0213da (0x4817 bytes) 000298 1787864224.576261 read 0x010dcd thru 0x01d59c (0xc7d0 bytes) 000299 1787864224.598174 read 0x018487 thru 0x01dac6 (0x5640 bytes) 000300 1787864224.618730 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x213db): Mirror 5: 0xfad5319f (stale) Mirror 6: 0xfad5319f (stale) Mirror 7: 0xfad5319f (stale) Mirror 8: 0xfad5319f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x213db): Mirror 5: 0xfad5319f (stale) Mirror 6: 0xfad5319f (stale) Mirror 7: 0xfad5319f (stale) Mirror 8: 0xfad5319f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x213db: skipped 000301 1787864225.316865 read 0x01929c thru 0x0213da (0x813f bytes) 000306 1787864225.335814 trunc from 0x0213db to 0x007d88 (0x19652 bytes) 000307 1787864225.404223 MIRROR_RESYNC 5 mirrors 000309 1787864226.731740 write 0x004386 thru 0x00ef07 (0xab82 bytes) 000314 1787864226.815458 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xef08): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xef08): Mirror 5: 0x20814cf0 (stale) Mirror 6: 0x20814cf0 (stale) Mirror 7: 0x20814cf0 (stale) Mirror 8: 0x20814cf0 (stale) CRC-32 checksum value for chunk [0, 0xef08): Mirror 1: 0x1f295a0 Mirror 9: 0x1f295a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xef08: skipped 000315 1787864227.872148 read 0x00623b thru 0x00ef07 (0x8ccd bytes) 000319 1787864227.896459 trunc from 0x00ef08 to 0x02b7ae (0x1c8a7 bytes) 000321 1787864228.116539 write 0x0270a7 thru 0x027732 (0x68c bytes) 000322 1787864228.127691 trunc from 0x02b7af to 0x003007 (0x287a7 bytes) 000323 1787864228.162968 write 0x03c9fe thru 0x03ffff (0x3602 bytes) 000324 1787864228.166917 mapread 0x029c72 thru 0x03086f (0x6bfe bytes) 000325 1787864228.213806 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbb7e2b0e (stale) Mirror 6: 0xbb7e2b0e (stale) Mirror 7: 0xbb7e2b0e (stale) Mirror 8: 0xbb7e2b0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbb7e2b0e (stale) Mirror 6: 0xbb7e2b0e (stale) Mirror 7: 0xbb7e2b0e (stale) Mirror 8: 0xbb7e2b0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000331 1787864228.833382 read 0x01efd9 thru 0x026ffa (0x8022 bytes) 000332 1787864228.858102 mapread 0x01aab1 thru 0x020906 (0x5e56 bytes) 000334 1787864228.907496 read 0x03035a thru 0x03a2c0 (0x9f67 bytes) 000336 1787864228.961158 trunc from 0x040000 to 0x00ffd3 (0x3002c bytes) 000337 1787864228.988341 mapwrite 0x010542 thru 0x01f381 (0xee40 bytes) 000338 1787864229.185096 MIRROR_RESYNC 5 mirrors 000339 1787864230.221076 mapwrite 0x008d49 thru 0x01709c (0xe354 bytes) 000340 1787864230.594757 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f382): Mirror 5: 0xd4924f26 (stale) Mirror 6: 0xd4924f26 (stale) Mirror 8: 0xd4924f26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f382): Mirror 5: 0xd4924f26 (stale) Mirror 6: 0xd4924f26 (stale) Mirror 8: 0xd4924f26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f382: skipped 000341 1787864231.168713 mapwrite 0x034cfd thru 0x03c933 (0x7c37 bytes) 000342 1787864231.305031 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c934): Mirror 5: 0x77407c5e (stale) Mirror 6: 0x77407c5e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c934): Mirror 5: 0x77407c5e (stale) Mirror 6: 0x77407c5e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c934: skipped 000344 1787864231.863803 write 0x010b57 thru 0x017b4f (0x6ff9 bytes) 000346 1787864231.898365 MIRROR_RESYNC 3 mirrors 000347 1787864233.327327 write 0x01fa56 thru 0x02cdee (0xd399 bytes) 000348 1787864233.486536 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c934): Mirror 6: 0x2dd3c0f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c934): Mirror 6: 0x2dd3c0f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c934: skipped 000349 1787864233.760933 MIRROR_RESYNC 2 mirrors 000351 1787864234.622738 mapwrite 0x016523 thru 0x01ca5c (0x653a bytes) 000353 1787864234.826684 mapwrite 0x01229a thru 0x0193f9 (0x7160 bytes) 000354 1787864234.905869 mapwrite 0x02d350 thru 0x036980 (0x9631 bytes) 000356 1787864235.000482 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) 000360 1787864235.182585 mapread 0x032ced thru 0x03c749 (0x9a5d bytes) 000361 1787864235.216229 read 0x009dfc thru 0x00f774 (0x5979 bytes) 000362 1787864235.228534 mapread 0x01ece2 thru 0x0243f1 (0x5710 bytes) 000364 1787864235.244479 mapread 0x015bf8 thru 0x01a1d2 (0x45db bytes) 000365 1787864235.256559 write 0x038cda thru 0x03f095 (0x63bc bytes) 000366 1787864235.258988 write 0x0368cc thru 0x03ffff (0x9734 bytes) 000367 1787864235.261215 mapwrite 0x00219a thru 0x009c72 (0x7ad9 bytes) 000369 1787864235.359232 trunc from 0x040000 to 0x01040a (0x2fbf5 bytes) 000370 1787864235.388220 write 0x01f875 thru 0x02f565 (0xfcf1 bytes) 000373 1787864235.392492 read 0x00644a thru 0x00d117 (0x6cce bytes) 000374 1787864235.393827 read 0x003294 thru 0x005174 (0x1ee1 bytes) 000376 1787864235.396200 mapread 0x0190ec thru 0x01d5d6 (0x44eb bytes) 000381 1787864235.409640 read 0x0029d9 thru 0x00a4de (0x7b06 bytes) 000383 1787864235.410962 mapwrite 0x0368a8 thru 0x03cc5a (0x63b3 bytes) 000384 1787864235.530633 mapwrite 0x03cbea thru 0x03ee58 (0x226f bytes) 000385 1787864235.701858 mapwrite 0x01230e thru 0x020c86 (0xe979 bytes) 000388 1787864235.779833 write 0x03eca0 thru 0x03ffff (0x1360 bytes) 000390 1787864235.785011 trunc from 0x040000 to 0x001c7f (0x3e380 bytes) 000395 1787864235.821709 mapwrite 0x003118 thru 0x00af2d (0x7e16 bytes) 000396 1787864235.930370 trunc from 0x00af2e to 0x0277b0 (0x1c883 bytes) 000399 1787864235.949798 mapwrite 0x0206f6 thru 0x02d1e5 (0xcaf0 bytes) truncating to largest ever: 0x3cf20 000402 1787864236.090043 trunc from 0x02d1e6 to 0x03cf1f (0xfd3a bytes) 000405 1787864236.110406 MIRROR_EXTEND to 2 mirrors 000410 1787864237.352592 mapread 0x0159d0 thru 0x01949c (0x3acd bytes) 000411 1787864237.410261 mapwrite 0x0299c8 thru 0x033dbd (0xa3f6 bytes) 000412 1787864237.680584 mapread 0x00e0c0 thru 0x00f534 (0x1475 bytes) 000413 1787864237.702303 trunc from 0x03cf20 to 0x0164ad (0x26a72 bytes) 000415 1787864237.726181 read 0x00499c thru 0x00c1fc (0x7861 bytes) 000416 1787864237.734919 write 0x01b52b thru 0x022992 (0x7468 bytes) 000417 1787864237.737553 write 0x0320bb thru 0x036fa1 (0x4ee7 bytes) 000418 1787864237.739651 mapread 0x02c5b8 thru 0x0301fa (0x3c43 bytes) 000419 1787864237.750570 write 0x01748c thru 0x02310a (0xbc7f bytes) 000420 1787864237.761757 write 0x004951 thru 0x011789 (0xce39 bytes) 000421 1787864237.763027 write 0x01c2ce thru 0x027d90 (0xbac3 bytes) 000422 1787864237.766359 mapread 0x02fd6b thru 0x0326ed (0x2983 bytes) 000423 1787864237.767909 read 0x00dc96 thru 0x01410c (0x6477 bytes) 000425 1787864237.770448 read 0x01a72e thru 0x021950 (0x7223 bytes) 000427 1787864237.772927 mapwrite 0x0043f3 thru 0x006ef7 (0x2b05 bytes) 000428 1787864237.859573 mapread 0x033230 thru 0x036fa1 (0x3d72 bytes) 000433 1787864237.871783 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) 000434 1787864238.115060 mapread 0x012abf thru 0x012ae4 (0x26 bytes) 000437 1787864238.127095 read 0x006d01 thru 0x006f4f (0x24f bytes) 000438 1787864238.136262 trunc from 0x036fa2 to 0x03be1c (0x4e7b bytes) 000439 1787864238.164974 mapwrite 0x0147a4 thru 0x023d30 (0xf58d bytes) 000440 1787864238.270574 read 0x012a70 thru 0x01bd28 (0x92b9 bytes) 000441 1787864238.272532 read 0x022a37 thru 0x027232 (0x47fc bytes) 000444 1787864238.288924 mapread 0x01ba33 thru 0x0223a8 (0x6976 bytes) 000445 1787864238.290278 trunc from 0x03be1d to 0x022c60 (0x191bc bytes) 000446 1787864238.314352 mapread 0x02104d thru 0x022c60 (0x1c14 bytes) 000447 1787864238.316415 mapwrite 0x033785 thru 0x035639 (0x1eb5 bytes) 000448 1787864238.396748 write 0x02ee7e thru 0x03339b (0x451e bytes) 000450 1787864238.404308 read 0x0324c4 thru 0x035639 (0x3176 bytes) 000451 1787864238.405254 trunc from 0x03563a to 0x00ed3c (0x268fd bytes) 000452 1787864238.430431 write 0x03fbae thru 0x03ffff (0x452 bytes) 000453 1787864238.431724 write 0x02ffd2 thru 0x032f45 (0x2f74 bytes) 000455 1787864238.445316 write 0x0000b8 thru 0x002f47 (0x2e90 bytes) 000457 1787864238.446990 trunc from 0x040000 to 0x0098d7 (0x36728 bytes) 000460 1787864238.471594 trunc from 0x0098d8 to 0x00f381 (0x5aaa bytes) 000464 1787864238.498034 write 0x004d8e thru 0x00ca63 (0x7cd6 bytes) 000465 1787864238.505558 mapwrite 0x015be7 thru 0x017b10 (0x1f2a bytes) 000471 1787864238.609814 read 0x013954 thru 0x017b10 (0x41bd bytes) 000472 1787864238.621780 write 0x023628 thru 0x02df11 (0xa8ea bytes) 000474 1787864238.626307 mapwrite 0x0302f0 thru 0x03eb26 (0xe837 bytes) 000475 1787864238.733247 trunc from 0x03eb27 to 0x03a071 (0x4ab5 bytes) 000477 1787864238.758411 MIRROR_EXTEND to 2 mirrors 000479 1787864239.485127 write 0x0172a6 thru 0x0205c0 (0x931b bytes) 000483 1787864239.590530 MIRROR_RESYNC 2 mirrors 000485 1787864240.873440 read 0x027639 thru 0x0276e7 (0xaf bytes) 000488 1787864240.881482 mapread 0x019919 thru 0x026c4b (0xd333 bytes) 000491 1787864240.924275 mapwrite 0x02368c thru 0x02f992 (0xc307 bytes) 000493 1787864241.121696 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) 000495 1787864241.384272 read 0x00e3b9 thru 0x017685 (0x92cd bytes) 000498 1787864241.428182 trunc from 0x03a072 to 0x010bb0 (0x294c1 bytes) 000499 1787864241.465011 mapwrite 0x015ced thru 0x0235b9 (0xd8cd bytes) 000502 1787864241.689362 mapwrite 0x024f02 thru 0x032ac4 (0xdbc3 bytes) 000507 1787864241.883782 read 0x00d75e thru 0x0134b2 (0x5d55 bytes) 000509 1787864241.903572 mapwrite 0x002980 thru 0x00566a (0x2ceb bytes) 000514 1787864241.958852 mapwrite 0x008204 thru 0x010d95 (0x8b92 bytes) 000516 1787864242.067708 write 0x0144ae thru 0x02419c (0xfcef bytes) 000517 1787864242.069342 trunc from 0x032ac5 to 0x011d99 (0x20d2b bytes) 000519 1787864242.132271 read 0x003381 thru 0x00aca5 (0x7925 bytes) 000521 1787864242.144394 mapread 0x007650 thru 0x011d99 (0xa74a bytes) 000524 1787864242.153909 mapwrite 0x03f6bb thru 0x03ffff (0x945 bytes) 000527 1787864242.321140 write 0x02d288 thru 0x0313e5 (0x415e bytes) 000528 1787864242.323091 write 0x022264 thru 0x0281c5 (0x5f62 bytes) 000531 1787864242.325221 read 0x0160cd thru 0x023dc7 (0xdcfb bytes) 000533 1787864242.326814 trunc from 0x040000 to 0x02ca64 (0x1359b bytes) 000534 1787864242.381067 mapwrite 0x01fc2a thru 0x020ba0 (0xf77 bytes) 000537 1787864242.468725 MIRROR_EXTEND to 2 mirrors 000543 1787864243.845092 mapread 0x008f0c thru 0x014664 (0xb759 bytes) 000547 1787864243.887077 MIRROR_EXTEND to 3 mirrors 000548 1787864245.064893 write 0x01e926 thru 0x022a8f (0x416a bytes) 000549 1787864245.139739 write 0x02216d thru 0x025a68 (0x38fc bytes) 000551 1787864245.145951 mapwrite 0x014f43 thru 0x018356 (0x3414 bytes) 000556 1787864245.357010 write 0x00a7e3 thru 0x013817 (0x9035 bytes) 000559 1787864245.379969 write 0x03e779 thru 0x03ffff (0x1887 bytes) 000562 1787864245.387524 mapwrite 0x018abc thru 0x01fbd6 (0x711b bytes) 000563 1787864245.517526 MIRROR_RESYNC 3 mirrors 000567 1787864246.624043 read 0x01af49 thru 0x01cf80 (0x2038 bytes) 000568 1787864246.629532 write 0x00f171 thru 0x01a23f (0xb0cf bytes) 000569 1787864246.740431 MIRROR_RESYNC 3 mirrors 000570 1787864247.804496 read 0x019311 thru 0x026cc4 (0xd9b4 bytes) 000572 1787864247.828245 write 0x016b39 thru 0x01c01f (0x54e7 bytes) 000574 1787864247.987566 write 0x022853 thru 0x02598e (0x313c bytes) 000575 1787864248.031320 write 0x019945 thru 0x029285 (0xf941 bytes) 000578 1787864248.058514 mapread 0x01bc9f thru 0x01d37a (0x16dc bytes) 000579 1787864248.059481 read 0x03f92b thru 0x03ffff (0x6d5 bytes) 000580 1787864248.104232 MIRROR_RESYNC 3 mirrors 000581 1787864249.031357 write 0x0098ea thru 0x00e259 (0x4970 bytes) 000582 1787864249.123314 write 0x02a6c1 thru 0x030d7e (0x66be bytes) 000583 1787864249.139166 trunc from 0x040000 to 0x02e5e8 (0x11a17 bytes) 000584 1787864249.204183 mapread 0x00fd5b thru 0x019a94 (0x9d3a bytes) 000585 1787864249.279442 read 0x011065 thru 0x011c40 (0xbdc bytes) 000586 1787864249.292293 MIRROR_RESYNC 3 mirrors 000587 1787864250.277898 mapread 0x02a226 thru 0x02e5e8 (0x43c3 bytes) 000588 1787864250.300768 trunc from 0x02e5e9 to 0x021a89 (0xcb5f bytes) 000589 1787864250.421257 write 0x0157bd thru 0x025499 (0xfcdd bytes) 000590 1787864250.429674 mapwrite 0x00ed17 thru 0x017e61 (0x914b bytes) 000592 1787864250.498943 read 0x024f4f thru 0x025499 (0x54b bytes) 000594 1787864250.499922 mapwrite 0x01e9a3 thru 0x028756 (0x9db4 bytes) 000595 1787864250.580838 trunc from 0x028757 to 0x006015 (0x22741 bytes) 000599 1787864250.608760 write 0x01b218 thru 0x02b198 (0xff81 bytes) 000600 1787864250.610851 read 0x018387 thru 0x01d441 (0x50bb bytes) 000602 1787864250.626185 read 0x020151 thru 0x022f33 (0x2de3 bytes) 000603 1787864250.627728 mapwrite 0x00a0d1 thru 0x00d477 (0x33a7 bytes) 000605 1787864250.685186 read 0x00a0d7 thru 0x00da42 (0x396c bytes) 000606 1787864250.690459 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2b199): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b199): Mirror 2: 0xd254f8b0 (stale) Mirror 3: 0xd254f8b0 (stale) CRC-32 checksum value for chunk [0, 0x2b199): Mirror 1: 0x45490383 Mirror 4: 0x45490383 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b199: skipped 000607 1787864251.820183 mapwrite 0x02b5c7 thru 0x02e30a (0x2d44 bytes) 000608 1787864252.122238 MIRROR_RESYNC 4 mirrors 000609 1787864253.208367 write 0x01432c thru 0x01890a (0x45df bytes) 000610 1787864253.336828 mapwrite 0x0052c4 thru 0x00ccd6 (0x7a13 bytes) 000611 1787864253.413049 write 0x003e93 thru 0x00e141 (0xa2af bytes) 000612 1787864253.414301 read 0x0148bd thru 0x023def (0xf533 bytes) 000614 1787864253.423646 write 0x023ac4 thru 0x02df96 (0xa4d3 bytes) 000615 1787864253.437539 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e30b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e30b): Mirror 2: 0x78ad3ba8 (stale) Mirror 3: 0x78ad3ba8 (stale) Mirror 4: 0x78ad3ba8 (stale) CRC-32 checksum value for chunk [0, 0x2e30b): Mirror 1: 0x5e0d8710 Mirror 5: 0x5e0d8710 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e30b: skipped 000617 1787864254.262139 mapwrite 0x00572b thru 0x012719 (0xcfef bytes) 000619 1787864254.567778 mapwrite 0x02d4ad thru 0x03b3d4 (0xdf28 bytes) 000625 1787864254.738681 mapwrite 0x0300ed thru 0x033cae (0x3bc2 bytes) 000626 1787864254.775306 read 0x00ec93 thru 0x011d5a (0x30c8 bytes) 000629 1787864254.778052 write 0x01cc4a thru 0x022985 (0x5d3c bytes) 000630 1787864254.789499 read 0x00a3ef thru 0x017a87 (0xd699 bytes) 000632 1787864254.795943 MIRROR_RESYNC 5 mirrors 000635 1787864256.407312 MIRROR_RESYNC 5 mirrors 000637 1787864256.996562 mapwrite 0x0146c2 thru 0x0168c9 (0x2208 bytes) 000638 1787864257.238197 trunc from 0x03b3d5 to 0x0115e2 (0x29df2 bytes) 000642 1787864257.278251 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x115e3): Mirror 2: 0x1eeadfdc (stale) Mirror 4: 0x1eeadfdc (stale) Mirror 5: 0x1eeadfdc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x115e3): Mirror 2: 0x1eeadfdc (stale) Mirror 4: 0x1eeadfdc (stale) Mirror 5: 0x1eeadfdc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x115e3: skipped 000645 1787864257.667302 write 0x03128d thru 0x03edde (0xdb52 bytes) 000646 1787864257.674831 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3eddf): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eddf): Mirror 2: 0x21df7965 (stale) Mirror 4: 0x21df7965 (stale) Mirror 5: 0x21df7965 (stale) CRC-32 checksum value for chunk [0, 0x3eddf): Mirror 1: 0x1804e3b Mirror 6: 0x1804e3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eddf: skipped 000649 1787864258.782344 trunc from 0x03eddf to 0x002396 (0x3ca48 bytes) 000650 1787864258.864970 mapread 0x001ff8 thru 0x002396 (0x39f bytes) 000651 1787864258.876523 mapread 0x001608 thru 0x002396 (0xd8f bytes) 000652 1787864258.880567 write 0x00c17e thru 0x014dda (0x8c5d bytes) 000654 1787864258.883372 trunc from 0x014ddb to 0x007e65 (0xcf75 bytes) 000655 1787864258.902406 mapwrite 0x0286a8 thru 0x02a4eb (0x1e44 bytes) 000656 1787864258.965536 read 0x00b5d5 thru 0x015032 (0x9a5e bytes) 000659 1787864258.992181 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2a4ec): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a4ec): Mirror 2: 0xe9393584 (stale) Mirror 4: 0xe9393584 (stale) Mirror 5: 0xe9393584 (stale) Mirror 6: 0x1ee27b52 (stale) CRC-32 checksum value for chunk [0, 0x2a4ec): Mirror 1: 0x5475b04f Mirror 7: 0x5475b04f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a4ec: skipped 000660 1787864260.270551 MIRROR_RESYNC 6 mirrors 000662 1787864261.576563 trunc from 0x02a4ec to 0x039f73 (0xfa88 bytes) 000663 1787864261.683369 trunc from 0x039f74 to 0x017e49 (0x2212a bytes) 000666 1787864261.718753 mapwrite 0x02c868 thru 0x03abc8 (0xe361 bytes) 000672 1787864261.823094 write 0x035d9c thru 0x03df5d (0x81c2 bytes) 000673 1787864261.824421 write 0x022f3a thru 0x0305f3 (0xd6ba bytes) 000674 1787864261.829077 read 0x038fd4 thru 0x03df5d (0x4f8a bytes) 000675 1787864261.829579 read 0x02bd8d thru 0x02cb3e (0xdb2 bytes) 000678 1787864261.831406 read 0x01f16a thru 0x023467 (0x42fe bytes) 000679 1787864261.870011 read 0x020b86 thru 0x02b164 (0xa5df bytes) 000680 1787864261.876922 MIRROR_RESYNC 6 mirrors 000681 1787864264.072982 read 0x00991a thru 0x0129ab (0x9092 bytes) 000685 1787864264.087281 mapwrite 0x02e96e thru 0x039cda (0xb36d bytes) 000686 1787864264.315550 mapread 0x01712b thru 0x019df7 (0x2ccd bytes) 000689 1787864264.339923 MIRROR_RESYNC 6 mirrors 000690 1787864266.018151 read 0x028846 thru 0x036780 (0xdf3b bytes) 000691 1787864266.035482 trunc from 0x03df5e to 0x0060fc (0x37e61 bytes) 000692 1787864266.262107 trunc from 0x0060fd to 0x023159 (0x1d05d bytes) 000693 1787864266.312786 mapread 0x010199 thru 0x01a54a (0xa3b2 bytes) truncating to largest ever: 0x3d2fb 000694 1787864266.360227 trunc from 0x02315a to 0x03d2fa (0x1a1a1 bytes) 000696 1787864266.407876 write 0x01cbe0 thru 0x02055e (0x397f bytes) 000697 1787864266.428286 write 0x039362 thru 0x03cdac (0x3a4b bytes) 000698 1787864266.446500 mapwrite 0x03f425 thru 0x03ffff (0xbdb bytes) 000700 1787864266.565044 trunc from 0x040000 to 0x02fa31 (0x105ce bytes) 000701 1787864266.589420 MIRROR_RESYNC 6 mirrors 000702 1787864267.672375 trunc from 0x02fa32 to 0x03acba (0xb289 bytes) 000703 1787864267.798258 mapwrite 0x0071aa thru 0x00f282 (0x80d9 bytes) 000704 1787864267.946826 write 0x02cf43 thru 0x030b9c (0x3c5a bytes) 000705 1787864267.967711 mapread 0x027df3 thru 0x03441a (0xc628 bytes) 000706 1787864268.002331 mapwrite 0x02fca7 thru 0x03ecfd (0xf057 bytes) 000707 1787864268.175565 write 0x01bb5c thru 0x01c9d3 (0xe78 bytes) 000709 1787864268.199185 trunc from 0x03ecfe to 0x039f19 (0x4de4 bytes) 000710 1787864268.235149 write 0x007bdc thru 0x014734 (0xcb59 bytes) 000711 1787864268.243382 write 0x036d7e thru 0x03c2e5 (0x5568 bytes) 000713 1787864268.246159 write 0x010dda thru 0x01be3a (0xb061 bytes) 000716 1787864268.250173 mapread 0x00fc50 thru 0x01d1d0 (0xd581 bytes) 000717 1787864268.259107 trunc from 0x03c2e6 to 0x01ff83 (0x1c362 bytes) 000718 1787864268.276705 mapread 0x004e78 thru 0x0139f4 (0xeb7d bytes) 000719 1787864268.281369 read 0x011d9a thru 0x01d19a (0xb401 bytes) 000726 1787864268.292151 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ff84): Mirror 4: 0xd2013aa8 (stale) Mirror 5: 0xd2013aa8 (stale) Mirror 6: 0xd2013aa8 (stale) Mirror 7: 0xd2013aa8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ff84): Mirror 4: 0xd2013aa8 (stale) Mirror 5: 0xd2013aa8 (stale) Mirror 6: 0xd2013aa8 (stale) Mirror 7: 0xd2013aa8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ff84: skipped 000727 1787864269.025446 mapread 0x01bd41 thru 0x01ff83 (0x4243 bytes) 000728 1787864269.055657 write 0x01b6c1 thru 0x02b242 (0xfb82 bytes) 000729 1787864269.057190 write 0x018d47 thru 0x025fcb (0xd285 bytes) 000730 1787864269.061949 mapread 0x028948 thru 0x02b242 (0x28fb bytes) 000732 1787864269.065965 write 0x039459 thru 0x03ffff (0x6ba7 bytes) 000733 1787864269.070890 mapread 0x02257c thru 0x023085 (0xb0a bytes) 000735 1787864269.077104 write 0x005f4b thru 0x00cd0d (0x6dc3 bytes) 000738 1787864269.116026 read 0x02a316 thru 0x02f8bb (0x55a6 bytes) 000739 1787864269.134451 mapwrite 0x00f90d thru 0x011dfb (0x24ef bytes) 000740 1787864269.313112 trunc from 0x040000 to 0x01f8d2 (0x2072d bytes) 000741 1787864269.363496 mapread 0x01cb26 thru 0x01cd4a (0x225 bytes) 000744 1787864269.373506 trunc from 0x01f8d3 to 0x01f2ca (0x608 bytes) 000745 1787864269.415066 mapread 0x01e9d0 thru 0x01f2ca (0x8fb bytes) 000746 1787864269.418711 trunc from 0x01f2cb to 0x005f3a (0x19390 bytes) 000751 1787864269.447832 mapwrite 0x00d762 thru 0x0162f4 (0x8b93 bytes) 000753 1787864269.587151 mapread 0x0059ad thru 0x00b521 (0x5b75 bytes) 000756 1787864269.591647 mapwrite 0x034028 thru 0x036452 (0x242b bytes) 000758 1787864269.760913 write 0x038c63 thru 0x03ffff (0x739d bytes) 000759 1787864269.777188 MIRROR_RESYNC 5 mirrors 000761 1787864272.156478 read 0x01517d thru 0x018bd5 (0x3a59 bytes) 000762 1787864272.162866 mapwrite 0x03c1ae thru 0x03fff2 (0x3e45 bytes) 000765 1787864272.321198 trunc from 0x040000 to 0x00ef83 (0x3107c bytes) 000766 1787864272.349955 mapwrite 0x02bdd3 thru 0x0312f4 (0x5522 bytes) 000767 1787864272.473547 MIRROR_RESYNC 5 mirrors 000769 1787864274.799455 mapread 0x011963 thru 0x012429 (0xac7 bytes) 000770 1787864274.841153 mapwrite 0x026b5d thru 0x033624 (0xcac8 bytes) 000771 1787864275.128582 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33625): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33625): Mirror 4: 0xa3d57ad3 (stale) Mirror 5: 0xa3d57ad3 (stale) Mirror 6: 0xa3d57ad3 (stale) Mirror 7: 0xa3d57ad3 (stale) CRC-32 checksum value for chunk [0, 0x33625): Mirror 1: 0x321bb249 Mirror 8: 0x321bb249 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33625: skipped 000772 1787864276.436709 mapwrite 0x03ea3f thru 0x03ffff (0x15c1 bytes) 000773 1787864276.658100 write 0x03f303 thru 0x03ffff (0xcfd bytes) 000774 1787864276.659270 trunc from 0x040000 to 0x036063 (0x9f9c bytes) 000775 1787864276.740525 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36064): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36064): Mirror 4: 0x6926d378 (stale) Mirror 5: 0x6926d378 (stale) Mirror 6: 0x6926d378 (stale) Mirror 7: 0x6926d378 (stale) Mirror 8: 0xce7473bc (stale) CRC-32 checksum value for chunk [0, 0x36064): Mirror 1: 0xce7473bc Mirror 9: 0xce7473bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36064: skipped 000777 1787864279.116596 mapread 0x0014ff thru 0x010b50 (0xf652 bytes) 000778 1787864279.179892 trunc from 0x036064 to 0x009f3d (0x2c126 bytes) 000779 1787864279.396590 read 0x0021ad thru 0x009f3d (0x7d91 bytes) 000780 1787864279.415739 trunc from 0x009f3e to 0x01f0f2 (0x151b5 bytes) 000781 1787864279.454424 write 0x01f1de thru 0x02d834 (0xe657 bytes) 000783 1787864279.464779 write 0x00e732 thru 0x00f375 (0xc44 bytes) 000784 1787864279.485004 write 0x015047 thru 0x018194 (0x314e bytes) 000786 1787864279.507754 MIRROR_RESYNC 7 mirrors 000788 1787864281.345025 mapread 0x011bdb thru 0x015c8c (0x40b2 bytes) 000790 1787864281.398165 MIRROR_RESYNC 7 mirrors 000791 1787864282.366362 trunc from 0x02d835 to 0x021cc8 (0xbb6c bytes) 000792 1787864282.572574 trunc from 0x021cc9 to 0x02d85e (0xbb96 bytes) 000796 1787864282.625795 write 0x019c3d thru 0x025eb5 (0xc279 bytes) 000798 1787864282.684273 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2d85f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d85f): Mirror 4: 0x783e397e (stale) Mirror 5: 0x783e397e (stale) Mirror 6: 0x783e397e (stale) Mirror 7: 0x783e397e (stale) Mirror 8: 0x783e397e (stale) Mirror 9: 0x783e397e (stale) CRC-32 checksum value for chunk [0, 0x2d85f): Mirror 1: 0x6cda1153 Mirror 10: 0x6cda1153 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d85f: skipped 000799 1787864283.943457 mapread 0x018fd3 thru 0x019c9a (0xcc8 bytes) 000802 1787864283.967494 trunc from 0x02d85f to 0x021d04 (0xbb5a bytes) 000803 1787864284.139962 write 0x0251a0 thru 0x02a1b3 (0x5014 bytes) 000804 1787864284.141369 mapwrite 0x013c05 thru 0x015364 (0x1760 bytes) 000811 1787864284.263042 mapwrite 0x03afee thru 0x03ffff (0x5012 bytes) 000812 1787864284.442134 read 0x02b65b thru 0x03a441 (0xede7 bytes) 000813 1787864284.466656 mapread 0x02981b thru 0x030112 (0x68f8 bytes) 000816 1787864284.468847 write 0x01ac76 thru 0x01d6c1 (0x2a4c bytes) 000817 1787864284.510318 read 0x03926b thru 0x03ffff (0x6d95 bytes) 000818 1787864284.511230 read 0x000286 thru 0x0004be (0x239 bytes) 000819 1787864284.551432 read 0x008c8c thru 0x00fbea (0x6f5f bytes) 000820 1787864284.552394 write 0x036c8b thru 0x03e094 (0x740a bytes) 000821 1787864284.558852 read 0x006e6d thru 0x00d4c9 (0x665d bytes) 000822 1787864284.560691 mapread 0x0282d0 thru 0x035feb (0xdd1c bytes) 000823 1787864284.575165 write 0x00c901 thru 0x013cf9 (0x73f9 bytes) 000825 1787864284.579381 trunc from 0x040000 to 0x010739 (0x2f8c6 bytes) 000826 1787864284.618842 mapwrite 0x02aa60 thru 0x031761 (0x6d02 bytes) 000827 1787864284.780989 mapread 0x0153ce thru 0x01d490 (0x80c3 bytes) 000828 1787864284.783436 write 0x02cbb1 thru 0x02e840 (0x1c90 bytes) 000830 1787864284.786051 mapwrite 0x034995 thru 0x03fb77 (0xb1e3 bytes) 000831 1787864284.914842 MIRROR_RESYNC 8 mirrors 000832 1787864287.925354 mapread 0x0310c4 thru 0x038388 (0x72c5 bytes) 000833 1787864287.946261 read 0x008244 thru 0x00f544 (0x7301 bytes) 000834 1787864287.960833 MIRROR_RESYNC 8 mirrors 000836 1787864288.461715 trunc from 0x03fb78 to 0x034b93 (0xafe4 bytes) 000838 1787864288.551787 mapwrite 0x034230 thru 0x03f8db (0xb6ac bytes) 000839 1787864288.670085 read 0x02fa46 thru 0x03cbe3 (0xd19e bytes) 000840 1787864288.671097 write 0x01337f thru 0x01d2ef (0x9f71 bytes) 000842 1787864288.684157 trunc from 0x03f8dc to 0x008a02 (0x36ed9 bytes) 000844 1787864288.713083 mapread 0x008789 thru 0x008a02 (0x27a bytes) 000845 1787864288.731049 trunc from 0x008a03 to 0x02683c (0x1de3a bytes) 000848 1787864288.757444 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2683d): Mirror 4: 0x9da269f6 (stale) Mirror 5: 0x9da269f6 (stale) Mirror 6: 0x9da269f6 (stale) Mirror 7: 0x9da269f6 (stale) Mirror 9: 0x9da269f6 (stale) Mirror 10: 0x9da269f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2683d): Mirror 4: 0x9da269f6 (stale) Mirror 5: 0x9da269f6 (stale) Mirror 6: 0x9da269f6 (stale) Mirror 7: 0x9da269f6 (stale) Mirror 9: 0x9da269f6 (stale) Mirror 10: 0x9da269f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2683d: skipped 000849 1787864289.326092 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2683d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2683d): Mirror 4: 0x9da269f6 (stale) Mirror 5: 0x9da269f6 (stale) Mirror 6: 0x9da269f6 (stale) Mirror 7: 0x9da269f6 (stale) Mirror 9: 0x9da269f6 (stale) Mirror 10: 0x9da269f6 (stale) CRC-32 checksum value for chunk [0, 0x2683d): Mirror 1: 0x7f8e48c4 Mirror 11: 0x7f8e48c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2683d: skipped 000850 1787864291.378943 write 0x02331f thru 0x026bdb (0x38bd bytes) 000853 1787864291.583358 MIRROR_RESYNC 8 mirrors 000855 1787864293.540338 write 0x00255c thru 0x0093e0 (0x6e85 bytes) 000856 1787864293.615498 trunc from 0x026bdc to 0x0359ef (0xee14 bytes) 000857 1787864293.640339 write 0x036739 thru 0x03ac52 (0x451a bytes) 000858 1787864293.643027 mapread 0x036837 thru 0x03ac52 (0x441c bytes) 000861 1787864293.653350 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ac53): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac53): Mirror 4: 0x2b8290a2 (stale) Mirror 5: 0x2b8290a2 (stale) Mirror 6: 0x2b8290a2 (stale) Mirror 7: 0x2b8290a2 (stale) Mirror 9: 0x2b8290a2 (stale) Mirror 10: 0x2b8290a2 (stale) Mirror 11: 0x2b8290a2 (stale) CRC-32 checksum value for chunk [0, 0x3ac53): Mirror 1: 0xdaf39be2 Mirror 12: 0xdaf39be2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac53: skipped 000862 1787864295.530453 write 0x0102e8 thru 0x015b9a (0x58b3 bytes) 000865 1787864295.863594 MIRROR_RESYNC 9 mirrors 000869 1787864298.573582 write 0x02721e thru 0x0295b4 (0x2397 bytes) 000870 1787864298.686419 read 0x00d584 thru 0x017a96 (0xa513 bytes) 000871 1787864298.701109 read 0x02eb8c thru 0x030c3a (0x20af bytes) 000872 1787864298.711833 write 0x00cb08 thru 0x01ab4b (0xe044 bytes) 000873 1787864298.713349 mapread 0x028935 thru 0x02e0f5 (0x57c1 bytes) 000875 1787864298.715645 write 0x01499f thru 0x0186e5 (0x3d47 bytes) 000877 1787864298.717290 read 0x00ab19 thru 0x00d572 (0x2a5a bytes) 000878 1787864298.718953 mapread 0x00a957 thru 0x012de4 (0x848e bytes) 000879 1787864298.721599 write 0x019c25 thru 0x022317 (0x86f3 bytes) 000880 1787864298.724418 trunc from 0x03ac53 to 0x0224a0 (0x187b2 bytes) 000881 1787864298.753337 read 0x01a1b7 thru 0x01d698 (0x34e2 bytes) 000883 1787864298.755347 write 0x0349c0 thru 0x0363e3 (0x1a24 bytes) 000887 1787864298.759077 mapread 0x0226ab thru 0x029e52 (0x77a8 bytes) 000889 1787864298.783090 mapwrite 0x024be1 thru 0x029673 (0x4a93 bytes) 000890 1787864298.840567 trunc from 0x0363e4 to 0x03a2dd (0x3efa bytes) 000891 1787864298.866851 MIRROR_RESYNC 9 mirrors 000892 1787864300.536180 MIRROR_EXTEND to 10 mirrors 000893 1787864301.565106 trunc from 0x03a2de to 0x02621f (0x140be bytes) 000895 1787864301.676418 write 0x03cad7 thru 0x03d4c9 (0x9f3 bytes) 000896 1787864301.680660 mapread 0x00eec0 thru 0x01b109 (0xc24a bytes) 000897 1787864301.720537 mapread 0x003eae thru 0x012bf7 (0xed4a bytes) 000898 1787864301.738473 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d4ca): Mirror 4: 0xbfe574b9 (stale) Mirror 5: 0xbfe574b9 (stale) Mirror 7: 0xbfe574b9 (stale) Mirror 9: 0xbfe574b9 (stale) Mirror 10: 0xbfe574b9 (stale) Mirror 11: 0xbfe574b9 (stale) Mirror 12: 0xbfe574b9 (stale) Mirror 13: 0xbfe574b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4ca): Mirror 4: 0xbfe574b9 (stale) Mirror 5: 0xbfe574b9 (stale) Mirror 7: 0xbfe574b9 (stale) Mirror 9: 0xbfe574b9 (stale) Mirror 10: 0xbfe574b9 (stale) Mirror 11: 0xbfe574b9 (stale) Mirror 12: 0xbfe574b9 (stale) Mirror 13: 0xbfe574b9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4ca: skipped 000901 1787864302.729548 mapwrite 0x023ff5 thru 0x033ea3 (0xfeaf bytes) 000902 1787864302.909685 write 0x02a9cb thru 0x03a16b (0xf7a1 bytes) 000903 1787864302.937768 MIRROR_RESYNC 9 mirrors 000904 1787864304.892976 mapread 0x023439 thru 0x02f4a2 (0xc06a bytes) 000905 1787864304.944246 MIRROR_RESYNC 9 mirrors 000906 1787864305.654397 read 0x0251a6 thru 0x0271e5 (0x2040 bytes) 000907 1787864305.663302 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) 000908 1787864306.196343 mapwrite 0x009c29 thru 0x00b5af (0x1987 bytes) 000911 1787864306.424448 mapread 0x01a936 thru 0x026832 (0xbefd bytes) 000912 1787864306.448422 trunc from 0x03d4ca to 0x02006b (0x1d45e bytes) 000913 1787864306.492297 write 0x00c45d thru 0x0131bc (0x6d60 bytes) 000914 1787864306.494314 trunc from 0x02006c to 0x020d23 (0xcb8 bytes) 000916 1787864306.527437 MIRROR_RESYNC 8 mirrors 000917 1787864307.935780 write 0x02a07a thru 0x02f75e (0x56e5 bytes) 000919 1787864308.040459 write 0x01c9cc thru 0x0296de (0xcd13 bytes) 000920 1787864308.063149 MIRROR_RESYNC 8 mirrors 000921 1787864310.599704 mapwrite 0x02fb07 thru 0x034d4d (0x5247 bytes) 000922 1787864310.979376 read 0x034252 thru 0x034d4d (0xafc bytes) 000923 1787864310.984684 trunc from 0x034d4e to 0x03593d (0xbf0 bytes) 000924 1787864311.049853 trunc from 0x03593e to 0x036b93 (0x1256 bytes) 000926 1787864311.080283 trunc from 0x036b94 to 0x008928 (0x2e26b bytes) 000928 1787864311.130398 read 0x0055e1 thru 0x008928 (0x3348 bytes) 000930 1787864311.167006 mapwrite 0x03f641 thru 0x03ffff (0x9bf bytes) 000931 1787864311.417426 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x974503b1 (stale) Mirror 5: 0x974503b1 (stale) Mirror 7: 0x974503b1 (stale) Mirror 9: 0x974503b1 (stale) Mirror 10: 0x974503b1 (stale) Mirror 11: 0x974503b1 (stale) Mirror 13: 0x974503b1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xabbdfa3 Mirror 14: 0xabbdfa3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000934 1787864313.416460 mapread 0x03c2b8 thru 0x03ffff (0x3d48 bytes) 000935 1787864313.475035 mapwrite 0x01795a thru 0x01f7c2 (0x7e69 bytes) 000937 1787864313.837025 write 0x00db9b thru 0x01d8f9 (0xfd5f bytes) 000938 1787864313.855527 mapread 0x030d23 thru 0x037551 (0x682f bytes) 000939 1787864313.884825 read 0x016de3 thru 0x025bdc (0xedfa bytes) 000943 1787864313.916669 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x185:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x974503b1 (stale) Mirror 5: 0x974503b1 (stale) Mirror 7: 0x974503b1 (stale) Mirror 9: 0x974503b1 (stale) Mirror 10: 0x974503b1 (stale) Mirror 11: 0x974503b1 (stale) Mirror 14: 0xabbdfa3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x974503b1 (stale) Mirror 5: 0x974503b1 (stale) Mirror 7: 0x974503b1 (stale) Mirror 9: 0x974503b1 (stale) Mirror 10: 0x974503b1 (stale) Mirror 11: 0x974503b1 (stale) Mirror 14: 0xabbdfa3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000944 1787864314.601304 write 0x00cbbe thru 0x01c85d (0xfca0 bytes) 000945 1787864314.627399 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x974503b1 (stale) Mirror 5: 0x974503b1 (stale) Mirror 7: 0x974503b1 (stale) Mirror 9: 0x974503b1 (stale) Mirror 10: 0x974503b1 (stale) Mirror 11: 0x974503b1 (stale) Mirror 14: 0xabbdfa3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6ffd81cc Mirror 15: 0x6ffd81cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000946 1787864316.272807 MIRROR_RESYNC 9 mirrors 000947 1787864318.257387 mapwrite 0x01857d thru 0x023973 (0xb3f7 bytes) 000948 1787864318.428486 trunc from 0x040000 to 0x019a64 (0x2659b bytes) 000949 1787864318.456513 read 0x0079de thru 0x009d94 (0x23b7 bytes) 000951 1787864318.463991 trunc from 0x019a65 to 0x001c46 (0x17e1e bytes) 000952 1787864318.481821 trunc from 0x001c47 to 0x033f50 (0x3230a bytes) 000953 1787864318.500256 write 0x00be9d thru 0x00e42a (0x258e bytes) 000954 1787864318.513553 read 0x010e0d thru 0x01c556 (0xb74a bytes) 000955 1787864318.531244 mapwrite 0x011b19 thru 0x02106d (0xf555 bytes) 000956 1787864318.594387 write 0x0328dc thru 0x03ffff (0xd724 bytes) 000957 1787864318.603895 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6ffd81cc (stale) Mirror 5: 0x6ffd81cc (stale) Mirror 7: 0x6ffd81cc (stale) Mirror 9: 0x6ffd81cc (stale) Mirror 10: 0x6ffd81cc (stale) Mirror 11: 0x6ffd81cc (stale) Mirror 14: 0x6ffd81cc (stale) Mirror 15: 0x6ffd81cc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96f686a8 Mirror 16: 0x96f686a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000962 1787864319.716982 mapwrite 0x03c217 thru 0x03f5ca (0x33b4 bytes) 000963 1787864319.933537 read 0x00c176 thru 0x00ce89 (0xd14 bytes) 000964 1787864319.947904 MIRROR_RESYNC 10 mirrors 000965 1787864322.082821 write 0x014e6b thru 0x015491 (0x627 bytes) 000968 1787864322.248694 mapwrite 0x0286ec thru 0x036606 (0xdf1b bytes) 000969 1787864322.418423 MIRROR_RESYNC 10 mirrors 000970 1787864325.349684 mapread 0x02a879 thru 0x036ddd (0xc565 bytes) 000974 1787864325.417958 write 0x036720 thru 0x037881 (0x1162 bytes) 000977 1787864325.665943 read 0x019d0a thru 0x02536e (0xb665 bytes) 000978 1787864325.700957 trunc from 0x040000 to 0x02fae1 (0x1051e bytes) 000979 1787864325.733804 mapread 0x026723 thru 0x02fae1 (0x93bf bytes) 000980 1787864325.750930 trunc from 0x02fae2 to 0x0032e5 (0x2c7fc bytes) 000982 1787864325.769795 read 0x0004a5 thru 0x0032e5 (0x2e41 bytes) 000983 1787864325.780816 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x32e6): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e6): Mirror 4: 0x2b5a2c1c (stale) Mirror 5: 0x2b5a2c1c (stale) Mirror 7: 0x2b5a2c1c (stale) Mirror 9: 0x2b5a2c1c (stale) Mirror 10: 0x2b5a2c1c (stale) Mirror 11: 0x2b5a2c1c (stale) Mirror 14: 0x2b5a2c1c (stale) Mirror 15: 0x2b5a2c1c (stale) Mirror 16: 0x2b5a2c1c (stale) CRC-32 checksum value for chunk [0, 0x32e6): Mirror 1: 0x2b5a2c1c Mirror 17: 0x2b5a2c1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e6: skipped 000985 1787864327.310019 MIRROR_RESYNC 11 mirrors 000986 1787864329.516407 MIRROR_RESYNC 11 mirrors 000989 1787864330.410966 mapwrite 0x03e660 thru 0x03ffff (0x19a0 bytes) 000990 1787864330.554857 read 0x03e017 thru 0x03ffff (0x1fe9 bytes) 000991 1787864330.559362 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2514da7f (stale) Mirror 5: 0x2514da7f (stale) Mirror 7: 0x2514da7f (stale) Mirror 9: 0x2514da7f (stale) Mirror 10: 0x2514da7f (stale) Mirror 11: 0x2514da7f (stale) Mirror 14: 0x2514da7f (stale) Mirror 15: 0x2514da7f (stale) Mirror 16: 0x2514da7f (stale) Mirror 17: 0x2514da7f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc9ee93dd Mirror 18: 0xc9ee93dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000992 1787864332.286424 write 0x01fc0d thru 0x02156f (0x1963 bytes) 000994 1787864332.551425 mapwrite 0x0303a2 thru 0x03efbc (0xec1b bytes) 000995 1787864332.766749 mapwrite 0x022402 thru 0x02dac1 (0xb6c0 bytes) 000996 1787864332.981716 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x2514da7f (stale) Mirror 5: 0x2514da7f (stale) Mirror 7: 0x2514da7f (stale) Mirror 9: 0x2514da7f (stale) Mirror 10: 0x2514da7f (stale) Mirror 11: 0x2514da7f (stale) Mirror 14: 0x2514da7f (stale) Mirror 15: 0x2514da7f (stale) Mirror 16: 0x2514da7f (stale) Mirror 17: 0x2514da7f (stale) Mirror 18: 0xc9ee93dd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xea50c5a4 Mirror 19: 0xea50c5a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000997 1787864335.499045 MIRROR_RESYNC 13 mirrors All operations completed A-OK! PASS 70a (145s) == sanity-flr test 71: check mirror extend parent fid ==== 16:59:04 (1787864344) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 16:59:09 (1787864349) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.217331 s, 9.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217732 s, 4.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 16:59:20 (1787864360) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1800288 .. Extending file size to 2834016 .. failed Extending file size to 4541056 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8750816 .. Extending file size to 5324960 .. Extending file size to 8581376 .. Extending file size to 9165280 .. Extending file size to 8768064 .. Extending file size to 8885376 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58620 58621 58622 58623 58625 done 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) 192.168.201.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:00:41 (1787864441) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1948672 .. Extending file size to 8780608 .. Extending file size to 4751968 .. Extending file size to 4831776 .. Extending file size to 5702336 .. Extending file size to 5973536 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6587840 .. Extending file size to 8241600 .. Extending file size to 8876256 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8993696 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9217504 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60174 60175 60178 60179 60181 done 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) 192.168.201.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:02:02 (1787864522) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158568 s, 13.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.349479 s, 6.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x398771f4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:02:25 (1787864545) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100841 s, 20.8 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 17:02:34 (1787864554) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1c6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:02:41 (1787864561) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.232391 s, 9.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:02:50 (1787864570) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.225403 s, 9.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:03:00 (1787864580) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173023 s, 12.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:03:35 (1787864615) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128694 s, 8.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:04:40 (1787864680) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.341384 s, 6.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:05:43 (1787864743) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. Creating new pool test_204f oleg138-server: Pool lustre.test_204f created Adding targets to pool oleg138-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg138-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.416523 s, 5.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg138-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg138-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg138-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:07:08 (1787864828) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:07:16 (1787864836) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:07:24 (1787864844) oleg138-server: Pool lustre.test_206 created Adding targets to pool oleg138-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg138-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg138-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg138-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg138-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:07:53 (1787864873) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111656 s, 9.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415301 s, 25.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x62bdf9c Mirror 3: 0x62bdf9c PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:08:05 (1787864885) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:08:09 (1787864889) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:08:13 (1787864893) Starting client: oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.13967 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x9b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1f80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1f80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1f80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2320000 Read this (4 bytes): BBBB 192.168.201.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg138-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:08:53 (1787864933) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0335506 s, 244 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:09:02 (1787864942) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135752 s, 7.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1eb lmm_fid: [0x200000404:0x1eb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1eb lmm_fid: [0x200000404:0x1eb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834411 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 17:09:04.000000000 -0400 Modify: 2026-08-27 17:09:05.000000000 -0400 Change: 2026-08-27 17:09:07.000000000 -0400 Birth: 2026-08-27 17:09:04.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:09:14 (1787864954) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:09:18 (1787864958) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.80883 s, 50.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0493268 s, 83.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 17:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 17:09 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:09:28 (1787864968) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:09:37 (1787864977) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1960 sec =========== 17:09:47 (1787864987) === sanity-flr: start cleanup 17:09:48 (1787864988) === === sanity-flr: finish cleanup 17:09:55 (1787864995) ===