-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 16:57:43 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg214-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 16:58:05 (1787864285) === oleg214-client.virtnet: executing check_config_client /mnt/lustre oleg214-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg214-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8fbdc859b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8fbdc859b800.idle_timeout=debug disable quota as required oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 16:58:45 (1787864325) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7677] == sanity-flr test 0a: lfs mirror create with -N option == 16:58:48 (1787864328) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:59:08 (1787864348) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:59:12 (1787864352) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:59:16 (1787864356) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:59:45 (1787864385) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:59:50 (1787864390) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 16:59:54 (1787864394) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8fbdc859b800.stats=clear osc.lustre-OST0001-osc-ffff8fbdc859b800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:00:04 (1787864404) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:00:16 (1787864416) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:00:25 (1787864425) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 17:00:29 (1787864429) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:00:37 (1787864437) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 17:00:47 (1787864447) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:00:55 (1787864455) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:00:59 (1787864459) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:01:08 (1787864468) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:01:20 (1787864480) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:01:28 (1787864488) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 3.01789 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101966 s, 10.3 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:01:43 (1787864503) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.410515 s, 25.5 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 17:01:53 (1787864513) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:02:05 (1787864525) Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fbdc859b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fbdc859b800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8fbdc859b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fbdc859b800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg214-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:03:25 (1787864605) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.420774 s, 15.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fbdc859b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fbdc859b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:04:12 (1787864652) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fbdc859b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fbdc859b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8fbdc859b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fbdc859b800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg214-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:05:40 (1787864740) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.5124 s, 11.6 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 17:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.034s user 0m0.000s sys 0m0.194s Stopping /mnt/lustre-ost1 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8fbdc859b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8fbdc859b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.400s user 0m0.001s sys 0m0.186s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg214-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg214-server oleg214-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8fbdc859b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fbdc859b800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.048s user 0m0.001s sys 0m0.178s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg214-server: oleg214-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg214-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg214-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8fbdc859b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8fbdc859b800.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:07:11 (1787864831) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:07:14 (1787864834) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:07:17 (1787864837) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 17:07:20 (1787864840) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 17:07:27 (1787864847) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.453164 s, 27.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:07:39 (1787864859) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340037 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245389 s, 42.7 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:07:47 (1787864867) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:07:55 (1787864875) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00275883 s, 371 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114376 s, 895 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00436236 s, 235 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0021285 s, 481 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 17:08:07 (1787864887) create test files with size 15758047 .. 57a3732bf376606ba8c809f0504f2ab5 - bdd1fd51009a4358ebd6d67a6ad79abe - ff0a05392a9f110e048fafc9ea5c31b7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg214-client.virtnet /mnt/lustre (opts:) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 17:08:34 (1787864914) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:08:52 (1787864932) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:09:00 (1787864940) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139451 s, 7.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026291 s, 39.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 17:09:10 (1787864950) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181636 s, 11.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.414012 s, 10.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211699 s, 9.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 17:09:30 (1787864970) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 17:09:35 (1787864975) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:09:40 (1787864980) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403081 s, 26.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 17:09:49 (1787864989) llite.lustre-ffff8fbdc90d1800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 17:09:56 (1787864996) llite.lustre-ffff8fbdc90d1800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355989 s, 29.5 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026001 s, 40.3 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 17:10:05 (1787865005) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:10:08 (1787865008) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xea:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:10:19 (1787865019) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.273454 s, 38.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg214-client.virtnet /mnt/lustre (opts:) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:10:30 (1787865030) llite.lustre-ffff8fbdc2af3000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.343499 s, 30.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg214-client.virtnet /mnt/lustre (opts:) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8fbdc8106800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:10:44 (1787865044) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:11:23 (1787865083) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:12:02 (1787865122) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 17:12:05 (1787865125) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.205893 s, 15.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.155889 s, 20.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd709abab (nosync,stale) Mirror 2: 0xd709abab (nosync,stale) ** verify mirror 2 doesn't change original checksum: 084de68c9cb1ae853ce6a4a57b0b182f - mirror 1 checksum: f06e15bc05b48020b43f2205199f75c5 - mirror 2 checksum: 084de68c9cb1ae853ce6a4a57b0b182f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T17:12:08-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T17:12:08-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 084de68c9cb1ae853ce6a4a57b0b182f - mirror 1 checksum: f06e15bc05b48020b43f2205199f75c5 - mirror 2 checksum: f06e15bc05b48020b43f2205199f75c5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T17:12:08-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T17:12:08-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:12:17 (1787865137) Failing mds1 on oleg214-server Stopping /mnt/lustre-mds1 (opts:) on oleg214-server 17:12:24 (1787865144) shut down facet: mds1 facet_host: oleg214-server facet_failover_host: oleg214-server Failover mds1 to oleg214-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 17:12:57 (1787865177) targets are mounted 17:12:57 (1787865177) facet_failover done oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311575 s, 33.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:13:20 (1787865200) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:13:24 (1787865204) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:13:27 (1787865207) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00881345 s, 1.9 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:13:34 (1787865214) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:13:37 (1787865217) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00954983 s, 429 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.054268 s, 75.5 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:13:44 (1787865224) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:14:07 (1787865247) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:14:36 (1787865276) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:15:02 (1787865302) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:15:28 (1787865328) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 17:15:35 (1787865335) mirror create pid 55645 mirror split pid 55646 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55645 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55646 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (71s) == sanity-flr test 70a: flr mode fsx test ================ 17:16:47 (1787865407) Seed set to 5409 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1787865410.136891 write 0x0288af thru 0x029789 (0xedb bytes) 000005 1787865410.140386 read 0x0202ad thru 0x025f14 (0x5c68 bytes) 000006 1787865410.153318 write 0x001fa5 thru 0x002b77 (0xbd3 bytes) truncating to largest ever: 0x2b53f 000008 1787865410.168352 trunc from 0x02978a to 0x02b53e (0x1db5 bytes) 000010 1787865410.190296 trunc from 0x02b53f to 0x019410 (0x1212e bytes) 000011 1787865410.218801 trunc from 0x019411 to 0x01c3cb (0x2fbb bytes) 000013 1787865410.260503 trunc from 0x01c3cc to 0x0085be (0x13e0d bytes) 000017 1787865410.308077 MIRROR_EXTEND to 2 mirrors 000018 1787865411.995612 trunc from 0x0085bf to 0x001a9a (0x6b24 bytes) 000019 1787865412.191620 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000023 1787865412.504292 trunc from 0x001a9b to 0x019368 (0x178ce bytes) 000024 1787865412.522112 read 0x0032dd thru 0x003cac (0x9d0 bytes) 000027 1787865412.533065 MIRROR_EXTEND to 2 mirrors 000028 1787865413.570528 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000029 1787865413.839189 mapwrite 0x02390f thru 0x027621 (0x3d13 bytes) 000033 1787865414.164903 trunc from 0x027622 to 0x02891b (0x12fa bytes) 000034 1787865414.203924 mapread 0x00ae9a thru 0x011d1e (0x6e85 bytes) 000035 1787865414.232283 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x36866 000036 1787865415.946483 trunc from 0x02891c to 0x036865 (0xdf4a bytes) 000037 1787865416.070602 trunc from 0x036866 to 0x00fc37 (0x26c2e bytes) 000039 1787865416.120448 trunc from 0x00fc38 to 0x033008 (0x233d1 bytes) 000040 1787865416.152733 trunc from 0x033009 to 0x0285a4 (0xaa64 bytes) 000042 1787865416.205757 read 0x008814 thru 0x00d815 (0x5002 bytes) 000044 1787865416.221535 read 0x02023f thru 0x023f99 (0x3d5b bytes) 000045 1787865416.236829 MIRROR_RESYNC 2 mirrors 000046 1787865417.164450 mapwrite 0x00fc6c thru 0x0123f8 (0x278d bytes) 000047 1787865417.477325 read 0x023ac2 thru 0x0285a4 (0x4ae3 bytes) 000048 1787865417.497285 write 0x02b550 thru 0x031045 (0x5af6 bytes) 000049 1787865417.502597 write 0x0123b0 thru 0x01f617 (0xd268 bytes) 000050 1787865417.516645 read 0x022534 thru 0x02de18 (0xb8e5 bytes) 000051 1787865417.528751 read 0x02d262 thru 0x02fd62 (0x2b01 bytes) 000052 1787865417.530995 mapwrite 0x03cf71 thru 0x03ffff (0x308f bytes) 000053 1787865417.790500 mapread 0x011181 thru 0x018d9d (0x7c1d bytes) 000054 1787865417.791521 mapread 0x011f16 thru 0x01f6be (0xd7a9 bytes) 000058 1787865417.797804 mapread 0x02917c thru 0x03019b (0x7020 bytes) 000060 1787865417.804157 trunc from 0x040000 to 0x02f885 (0x1077a bytes) 000061 1787865417.851576 write 0x03cde8 thru 0x03ffff (0x3218 bytes) 000062 1787865417.853046 mapwrite 0x0266a5 thru 0x031a38 (0xb394 bytes) 000063 1787865418.073146 MIRROR_RESYNC 2 mirrors 000066 1787865419.191868 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000069 1787865419.411968 mapwrite 0x006573 thru 0x006d1d (0x7ab bytes) 000071 1787865419.476669 mapread 0x002155 thru 0x0034a5 (0x1351 bytes) 000074 1787865419.500737 trunc from 0x040000 to 0x0261b1 (0x19e4e bytes) 000075 1787865419.520571 mapread 0x01005b thru 0x01ef60 (0xef06 bytes) 000078 1787865419.540054 trunc from 0x0261b2 to 0x009e1e (0x1c393 bytes) 000081 1787865419.584241 trunc from 0x009e1f to 0x007b60 (0x22be bytes) truncating to largest ever: 0x372ef 000084 1787865419.628634 trunc from 0x007b61 to 0x0372ee (0x2f78e bytes) 000088 1787865419.651700 read 0x00f821 thru 0x016860 (0x7040 bytes) 000089 1787865419.669153 write 0x00601a thru 0x006e14 (0xdfb bytes) 000090 1787865419.670781 mapread 0x02a53b thru 0x03465f (0xa125 bytes) 000098 1787865419.751252 write 0x01f698 thru 0x0204b2 (0xe1b bytes) 000100 1787865419.792638 write 0x03a13e thru 0x03dc39 (0x3afc bytes) 000101 1787865419.798578 read 0x02e7e6 thru 0x03b8d7 (0xd0f2 bytes) 000102 1787865419.819321 write 0x01c0e1 thru 0x02432a (0x824a bytes) 000103 1787865419.820638 mapwrite 0x02ee8c thru 0x0394bf (0xa634 bytes) 000104 1787865420.011866 mapwrite 0x038ac8 thru 0x03d606 (0x4b3f bytes) truncating to largest ever: 0x3e902 000105 1787865420.090288 trunc from 0x03dc3a to 0x03e901 (0xcc8 bytes) 000106 1787865420.115217 trunc from 0x03e902 to 0x0344a0 (0xa461 bytes) 000107 1787865420.139662 mapwrite 0x00256c thru 0x005632 (0x30c7 bytes) 000108 1787865420.182772 trunc from 0x0344a1 to 0x003112 (0x3138e bytes) 000109 1787865420.205515 read 0x000b01 thru 0x001a97 (0xf97 bytes) 000110 1787865420.206788 write 0x021ae8 thru 0x0283f5 (0x690e bytes) 000112 1787865420.209240 mapwrite 0x032162 thru 0x035755 (0x35f4 bytes) 000116 1787865420.307604 mapread 0x02af02 thru 0x032119 (0x7218 bytes) 000117 1787865420.309254 write 0x02c71b thru 0x0389b3 (0xc299 bytes) 000118 1787865420.312502 read 0x024fb8 thru 0x0251b1 (0x1fa bytes) 000119 1787865420.313757 mapread 0x03060a thru 0x0389b3 (0x83aa bytes) 000120 1787865420.315715 read 0x007e37 thru 0x00a1fe (0x23c8 bytes) 000121 1787865420.317431 mapread 0x006759 thru 0x012b42 (0xc3ea bytes) 000122 1787865420.319155 mapread 0x011719 thru 0x021649 (0xff31 bytes) 000123 1787865420.323588 write 0x00cebe thru 0x010985 (0x3ac8 bytes) 000124 1787865420.325248 mapwrite 0x03cd72 thru 0x03ffff (0x328e bytes) 000125 1787865420.462476 mapwrite 0x028133 thru 0x02e699 (0x6567 bytes) 000126 1787865420.516132 mapwrite 0x0180e7 thru 0x028059 (0xff73 bytes) 000128 1787865420.581560 mapread 0x02892c thru 0x033cab (0xb380 bytes) 000129 1787865420.582652 trunc from 0x040000 to 0x013a9d (0x2c562 bytes) 000131 1787865420.611150 write 0x012365 thru 0x01ca41 (0xa6dd bytes) 000132 1787865420.618864 write 0x0138d0 thru 0x016ac5 (0x31f6 bytes) 000134 1787865420.625298 mapread 0x01435a thru 0x01a645 (0x62ec bytes) 000137 1787865420.629022 read 0x00bbed thru 0x011da6 (0x61ba bytes) 000138 1787865420.630054 read 0x008172 thru 0x010c37 (0x8ac6 bytes) 000139 1787865420.631210 mapwrite 0x008ba4 thru 0x008ff2 (0x44f bytes) 000140 1787865420.722556 write 0x010e9f thru 0x0185df (0x7741 bytes) 000141 1787865420.726570 trunc from 0x01ca42 to 0x0296e0 (0xcc9f bytes) 000142 1787865420.765589 mapread 0x00ee37 thru 0x0125d0 (0x379a bytes) 000143 1787865420.767741 read 0x01c19e thru 0x020e6e (0x4cd1 bytes) 000145 1787865420.780668 read 0x005741 thru 0x010e72 (0xb732 bytes) 000147 1787865420.782225 mapread 0x01cff4 thru 0x028586 (0xb593 bytes) 000153 1787865420.791594 mapread 0x018f11 thru 0x01d097 (0x4187 bytes) 000156 1787865420.802641 write 0x01e270 thru 0x0275c0 (0x9351 bytes) 000157 1787865420.806834 mapwrite 0x01861b thru 0x02618c (0xdb72 bytes) 000160 1787865420.909701 read 0x01e75f thru 0x026ddd (0x867f bytes) 000161 1787865420.910733 write 0x03e385 thru 0x03ffff (0x1c7b bytes) 000163 1787865420.915265 mapwrite 0x012315 thru 0x01ac6e (0x895a bytes) 000167 1787865421.063272 mapread 0x0192dc thru 0x01ccef (0x3a14 bytes) 000168 1787865421.070502 mapread 0x01c317 thru 0x01d7ec (0x14d6 bytes) 000177 1787865421.084781 read 0x027775 thru 0x02b00c (0x3898 bytes) 000178 1787865421.103289 trunc from 0x040000 to 0x032de7 (0xd218 bytes) 000179 1787865421.150056 MIRROR_EXTEND to 2 mirrors 000181 1787865422.890085 mapread 0x025ea2 thru 0x027033 (0x1192 bytes) 000183 1787865422.911755 write 0x02c12b thru 0x02d288 (0x115e bytes) 000185 1787865423.011906 mapread 0x015fff thru 0x02102c (0xb02e bytes) 000188 1787865423.054996 trunc from 0x032de8 to 0x0192c5 (0x19b22 bytes) 000190 1787865423.093671 write 0x004d4c thru 0x012354 (0xd609 bytes) 000191 1787865423.101899 trunc from 0x0192c6 to 0x014dd0 (0x44f5 bytes) 000194 1787865423.127163 mapwrite 0x0247c9 thru 0x02a3f2 (0x5c2a bytes) 000195 1787865423.277806 trunc from 0x02a3f3 to 0x007bea (0x22808 bytes) 000196 1787865423.321452 trunc from 0x007beb to 0x02696b (0x1ed81 bytes) 000198 1787865423.344229 mapwrite 0x01d3ec thru 0x02b5bf (0xe1d4 bytes) 000199 1787865423.474080 mapread 0x01485c thru 0x01b19a (0x693f bytes) 000200 1787865423.494152 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000201 1787865423.740104 read 0x0080c9 thru 0x011588 (0x94c0 bytes) 000202 1787865423.780830 mapwrite 0x011b38 thru 0x016248 (0x4711 bytes) 000203 1787865423.930722 MIRROR_EXTEND to 2 mirrors 000205 1787865425.008669 MIRROR_EXTEND to 3 mirrors 000208 1787865426.218018 write 0x02a1d5 thru 0x035592 (0xb3be bytes) 000209 1787865426.382084 read 0x00d463 thru 0x019612 (0xc1b0 bytes) 000213 1787865426.401940 mapread 0x0220a4 thru 0x027e58 (0x5db5 bytes) 000214 1787865426.424568 mapread 0x01adda thru 0x02186d (0x6a94 bytes) 000215 1787865426.437905 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35593): Mirror 3: 0xfcb9df5e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35593): Mirror 3: 0xfcb9df5e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35593: skipped 000216 1787865427.060605 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35593): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35593): Mirror 3: 0xfcb9df5e (stale) CRC-32 checksum value for chunk [0, 0x35593): Mirror 1: 0xc6734212 Mirror 4: 0xc6734212 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35593: skipped 000217 1787865428.790782 mapwrite 0x03d8f6 thru 0x03ffff (0x270a bytes) 000219 1787865429.175990 mapread 0x02ae83 thru 0x039020 (0xe19e bytes) 000221 1787865429.229881 read 0x020c27 thru 0x022fb6 (0x2390 bytes) 000222 1787865429.244316 mapread 0x02c39d thru 0x03326f (0x6ed3 bytes) 000223 1787865429.245357 read 0x0318dc thru 0x03726e (0x5993 bytes) 000224 1787865429.253901 MIRROR_RESYNC 3 mirrors 000225 1787865430.956447 trunc from 0x040000 to 0x039039 (0x6fc6 bytes) 000227 1787865431.074503 trunc from 0x03903a to 0x00f694 (0x299a5 bytes) 000229 1787865431.107059 read 0x008e81 thru 0x00e50c (0x568c bytes) 000235 1787865431.141234 mapread 0x006ae5 thru 0x00bc25 (0x5141 bytes) 000236 1787865431.165052 mapread 0x00eea0 thru 0x00f694 (0x7f5 bytes) 000237 1787865431.184272 mapread 0x002ea3 thru 0x007d31 (0x4e8f bytes) 000238 1787865431.197345 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xf695): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf695): Mirror 3: 0x4e3c5701 (stale) Mirror 4: 0x4e3c5701 (stale) CRC-32 checksum value for chunk [0, 0xf695): Mirror 1: 0x4e3c5701 Mirror 5: 0x4e3c5701 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf695: skipped 000239 1787865432.563448 mapwrite 0x005709 thru 0x00cbf8 (0x74f0 bytes) 000240 1787865432.900986 read 0x0015b2 thru 0x00257b (0xfca bytes) 000242 1787865432.903251 read 0x00d394 thru 0x00f694 (0x2301 bytes) 000243 1787865432.918131 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xf695): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf695): Mirror 3: 0x4e3c5701 (stale) Mirror 4: 0x4e3c5701 (stale) Mirror 5: 0x4e3c5701 (stale) CRC-32 checksum value for chunk [0, 0xf695): Mirror 1: 0x8cbb808c Mirror 6: 0x8cbb808c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf695: skipped 000245 1787865434.067288 read 0x0097c1 thru 0x00f694 (0x5ed4 bytes) 000246 1787865434.107396 MIRROR_RESYNC 5 mirrors 000247 1787865435.468216 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000249 1787865435.864942 mapread 0x0003f7 thru 0x00519c (0x4da6 bytes) 000251 1787865435.894931 mapread 0x00956a thru 0x00c58f (0x3026 bytes) 000253 1787865435.897049 trunc from 0x00f695 to 0x003f17 (0xb77d bytes) 000254 1787865436.036768 trunc from 0x003f18 to 0x021043 (0x1d12c bytes) 000255 1787865436.063009 read 0x004d4f thru 0x008579 (0x382b bytes) 000257 1787865436.078839 mapread 0x000af0 thru 0x00d710 (0xcc21 bytes) 000258 1787865436.097846 mapread 0x019fa6 thru 0x021043 (0x709e bytes) 000259 1787865436.115832 mapread 0x00bcad thru 0x0111d0 (0x5524 bytes) 000260 1787865436.124076 read 0x00b1b8 thru 0x013194 (0x7fdd bytes) 000263 1787865436.128706 read 0x00a34f thru 0x00aaf8 (0x7aa bytes) 000264 1787865436.130971 mapread 0x00ea9b thru 0x016d0b (0x8271 bytes) 000265 1787865436.135934 write 0x012f90 thru 0x01f7a2 (0xc813 bytes) 000269 1787865436.141636 trunc from 0x021044 to 0x000541 (0x20b02 bytes) 000272 1787865436.169143 trunc from 0x000542 to 0x030462 (0x2ff21 bytes) 000273 1787865436.185635 write 0x01f582 thru 0x02006e (0xaed bytes) 000274 1787865436.199195 write 0x03d825 thru 0x03ffff (0x27db bytes) 000275 1787865436.201884 read 0x0320b0 thru 0x037b34 (0x5a85 bytes) 000277 1787865436.209903 read 0x005401 thru 0x00fee1 (0xaae1 bytes) 000278 1787865436.210580 read 0x01c76b thru 0x0272ce (0xab64 bytes) 000279 1787865436.216576 MIRROR_RESYNC 4 mirrors 000280 1787865438.000567 mapread 0x0049e5 thru 0x012fae (0xe5ca bytes) 000281 1787865438.037230 mapwrite 0x034173 thru 0x03607c (0x1f0a bytes) 000285 1787865438.273550 write 0x030cb6 thru 0x03c7cf (0xbb1a bytes) 000288 1787865438.277975 mapwrite 0x035fb4 thru 0x03ffff (0xa04c bytes) 000289 1787865438.343358 read 0x02c3ca thru 0x02ee65 (0x2a9c bytes) 000290 1787865438.351926 write 0x038b76 thru 0x03e27a (0x5705 bytes) 000291 1787865438.352968 write 0x009bf9 thru 0x00deba (0x42c2 bytes) 000292 1787865438.370692 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x44515354 (stale) Mirror 4: 0x44515354 (stale) Mirror 6: 0x44515354 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4e15ea8c Mirror 7: 0x4e15ea8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000293 1787865440.363128 write 0x0343ba thru 0x036cb5 (0x28fc bytes) 000296 1787865440.596058 write 0x019dfa thru 0x02863b (0xe842 bytes) 000297 1787865440.620834 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x44515354 (stale) Mirror 4: 0x44515354 (stale) Mirror 6: 0x44515354 (stale) Mirror 7: 0x4e15ea8c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdb8a7ddc Mirror 8: 0xdb8a7ddc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000298 1787865442.732207 MIRROR_RESYNC 6 mirrors 000301 1787865446.006153 MIRROR_RESYNC 6 mirrors 000302 1787865446.973258 mapread 0x02f113 thru 0x038cd5 (0x9bc3 bytes) 000303 1787865447.012883 trunc from 0x040000 to 0x01a1a8 (0x25e57 bytes) 000305 1787865447.130970 trunc from 0x01a1a9 to 0x000a7a (0x1972e bytes) 000306 1787865447.198319 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa7b): Mirror 3: 0xf5c75d35 (stale) Mirror 6: 0xf5c75d35 (stale) Mirror 7: 0xf5c75d35 (stale) Mirror 8: 0xf5c75d35 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa7b): Mirror 3: 0xf5c75d35 (stale) Mirror 6: 0xf5c75d35 (stale) Mirror 7: 0xf5c75d35 (stale) Mirror 8: 0xf5c75d35 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa7b: skipped 000307 1787865447.800330 read 0x0009ad thru 0x000a7a (0xce bytes) 000313 1787865447.826624 write 0x01407c thru 0x023ac3 (0xfa48 bytes) 000317 1787865447.838379 read 0x0171f0 thru 0x023ac3 (0xc8d4 bytes) 000318 1787865447.839334 mapwrite 0x01caf2 thru 0x01d4a0 (0x9af bytes) 000319 1787865447.944697 read 0x0115a7 thru 0x018d5f (0x77b9 bytes) 000321 1787865447.967285 write 0x0271a3 thru 0x02c7b0 (0x560e bytes) 000322 1787865447.970232 mapread 0x022033 thru 0x022af9 (0xac7 bytes) 000325 1787865447.978538 trunc from 0x02c7b1 to 0x016e23 (0x1598d bytes) 000326 1787865448.009276 trunc from 0x016e24 to 0x00efc2 (0x7e61 bytes) 000327 1787865448.053863 write 0x02a19d thru 0x0306ed (0x6551 bytes) 000328 1787865448.061029 read 0x012c40 thru 0x01fd2d (0xd0ee bytes) 000333 1787865448.097933 read 0x006746 thru 0x00f633 (0x8eee bytes) 000334 1787865448.099445 write 0x013273 thru 0x01ab63 (0x78f1 bytes) 000339 1787865448.104974 write 0x016692 thru 0x0174c6 (0xe35 bytes) 000340 1787865448.107220 mapwrite 0x003394 thru 0x010721 (0xd38e bytes) 000342 1787865448.191353 write 0x00b327 thru 0x00d09d (0x1d77 bytes) 000345 1787865448.199956 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x306ee): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x306ee): Mirror 3: 0x686e4baf (stale) Mirror 6: 0x686e4baf (stale) Mirror 7: 0x686e4baf (stale) Mirror 8: 0x686e4baf (stale) CRC-32 checksum value for chunk [0, 0x306ee): Mirror 1: 0x344a05b9 Mirror 9: 0x344a05b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x306ee: skipped 000347 1787865449.507182 read 0x023206 thru 0x023981 (0x77c bytes) 000349 1787865449.512818 mapread 0x014974 thru 0x02284e (0xdedb bytes) 000350 1787865449.542729 read 0x010ef1 thru 0x01bc79 (0xad89 bytes) 000354 1787865449.544021 write 0x032c07 thru 0x03f5b3 (0xc9ad bytes) 000355 1787865449.646124 mapwrite 0x027995 thru 0x02b82f (0x3e9b bytes) 000357 1787865449.715083 read 0x02ad18 thru 0x03869c (0xd985 bytes) 000358 1787865449.715990 read 0x03e045 thru 0x03f5b3 (0x156f bytes) 000360 1787865449.717100 mapwrite 0x025549 thru 0x034aa6 (0xf55e bytes) 000361 1787865449.775510 mapread 0x03b6fb thru 0x03ed84 (0x368a bytes) 000363 1787865449.776947 mapwrite 0x031725 thru 0x034298 (0x2b74 bytes) 000364 1787865449.845814 read 0x02e359 thru 0x03436b (0x6013 bytes) 000366 1787865449.847298 mapwrite 0x00bb6a thru 0x00e0a3 (0x253a bytes) 000368 1787865449.923579 read 0x027710 thru 0x0365a8 (0xee99 bytes) 000371 1787865449.927812 trunc from 0x03f5b4 to 0x01b2a3 (0x24310 bytes) 000372 1787865449.976972 write 0x03b49c thru 0x03e309 (0x2e6e bytes) 000373 1787865449.989079 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e30a): Mirror 6: 0x1e268736 (stale) Mirror 7: 0x1e268736 (stale) Mirror 8: 0x1e268736 (stale) Mirror 9: 0x3b2e14da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e30a): Mirror 6: 0x1e268736 (stale) Mirror 7: 0x1e268736 (stale) Mirror 8: 0x1e268736 (stale) Mirror 9: 0x3b2e14da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e30a: skipped 000374 1787865450.735259 mapwrite 0x012140 thru 0x01d90b (0xb7cc bytes) 000375 1787865450.906284 trunc from 0x03e30a to 0x0004f6 (0x3de13 bytes) 000376 1787865450.944819 MIRROR_RESYNC 5 mirrors 000378 1787865452.749948 write 0x02d317 thru 0x03551c (0x8206 bytes) 000380 1787865452.855132 write 0x0344b8 thru 0x03ffff (0xbb48 bytes) 000382 1787865452.859330 mapwrite 0x037ec8 thru 0x03e337 (0x6470 bytes) 000383 1787865452.992574 read 0x018ec3 thru 0x020ea4 (0x7fe2 bytes) 000384 1787865453.013976 mapwrite 0x00557d thru 0x00a87c (0x5300 bytes) 000385 1787865453.117337 mapwrite 0x00e294 thru 0x0174c0 (0x922d bytes) 000387 1787865453.227040 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5dc5a960 Mirror 10: 0x5dc5a960 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000388 1787865455.119255 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5dc5a960 Mirror 10: 0x5dc5a960 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000391 1787865456.346806 mapread 0x0038cd thru 0x011a86 (0xe1ba bytes) 000392 1787865456.415129 mapread 0x005481 thru 0x007efa (0x2a7a bytes) 000394 1787865456.429975 MIRROR_RESYNC 5 mirrors 000396 1787865460.142906 mapread 0x03a7ca thru 0x03ffff (0x5836 bytes) 000397 1787865460.171537 trunc from 0x040000 to 0x017a72 (0x2858d bytes) 000398 1787865460.313171 read 0x0161ef thru 0x017589 (0x139b bytes) 000399 1787865460.329624 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x17a73): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17a73): Mirror 6: 0xed8ce3b0 (stale) Mirror 7: 0xed8ce3b0 (stale) Mirror 8: 0xed8ce3b0 (stale) Mirror 10: 0xed8ce3b0 (stale) CRC-32 checksum value for chunk [0, 0x17a73): Mirror 1: 0xed8ce3b0 Mirror 11: 0xed8ce3b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17a73: skipped 000401 1787865461.961391 mapwrite 0x00cd65 thru 0x01bd21 (0xefbd bytes) 000403 1787865462.457794 trunc from 0x01bd22 to 0x010487 (0xb89a bytes) 000405 1787865462.508683 MIRROR_RESYNC 6 mirrors 000406 1787865465.487087 mapread 0x0023d3 thru 0x00941f (0x704d bytes) 000407 1787865465.508745 mapwrite 0x019482 thru 0x01dd07 (0x4886 bytes) 000408 1787865465.911671 read 0x00f60b thru 0x0115f5 (0x1feb bytes) 000410 1787865465.936379 mapwrite 0x0318c7 thru 0x03f6e9 (0xde23 bytes) 000412 1787865466.132546 write 0x029426 thru 0x02ae40 (0x1a1b bytes) 000414 1787865466.149351 trunc from 0x03f6ea to 0x03c190 (0x3559 bytes) 000415 1787865466.178268 read 0x003e80 thru 0x005451 (0x15d2 bytes) 000416 1787865466.200801 read 0x03bd46 thru 0x03c190 (0x44b bytes) 000418 1787865466.204261 read 0x02d597 thru 0x039bde (0xc648 bytes) 000419 1787865466.206001 mapread 0x033ac1 thru 0x039d34 (0x6274 bytes) 000420 1787865466.207331 write 0x021480 thru 0x02c340 (0xaec1 bytes) 000421 1787865466.217969 write 0x03e27c thru 0x03ffff (0x1d84 bytes) 000425 1787865466.243885 read 0x029d27 thru 0x031e59 (0x8133 bytes) 000426 1787865466.245506 read 0x0066d4 thru 0x0163f6 (0xfd23 bytes) 000427 1787865466.261785 MIRROR_RESYNC 6 mirrors 000430 1787865469.564973 read 0x005f48 thru 0x007f7b (0x2034 bytes) 000432 1787865469.572139 mapwrite 0x00a333 thru 0x01a16d (0xfe3b bytes) 000434 1787865469.742024 MIRROR_RESYNC 6 mirrors 000435 1787865472.363783 write 0x017bc3 thru 0x01832f (0x76d bytes) 000436 1787865472.441322 MIRROR_RESYNC 6 mirrors 000437 1787865474.989244 write 0x03d365 thru 0x03ffff (0x2c9b bytes) 000438 1787865475.058694 mapwrite 0x02f62f thru 0x03c2e3 (0xccb5 bytes) 000443 1787865475.197056 trunc from 0x040000 to 0x0137dd (0x2c822 bytes) 000446 1787865475.250357 trunc from 0x0137de to 0x02f72f (0x1bf52 bytes) 000447 1787865475.278273 mapwrite 0x03a065 thru 0x03ce9c (0x2e38 bytes) 000448 1787865475.402137 mapwrite 0x01f6c1 thru 0x02b25d (0xbb9d bytes) 000450 1787865475.514066 write 0x002d46 thru 0x010a6e (0xdd29 bytes) 000451 1787865475.543148 write 0x0147f8 thru 0x0220b8 (0xd8c1 bytes) 000452 1787865475.559136 read 0x005375 thru 0x00fb22 (0xa7ae bytes) 000455 1787865475.566888 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ce9d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce9d): Mirror 6: 0x34680778 (stale) Mirror 7: 0x34680778 (stale) Mirror 8: 0x34680778 (stale) Mirror 10: 0x34680778 (stale) Mirror 11: 0x34680778 (stale) CRC-32 checksum value for chunk [0, 0x3ce9d): Mirror 1: 0x9ee328f7 Mirror 12: 0x9ee328f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce9d: skipped 000456 1787865477.098082 MIRROR_RESYNC 7 mirrors 000458 1787865479.002166 read 0x0114a4 thru 0x015bc9 (0x4726 bytes) 000461 1787865479.015559 MIRROR_RESYNC 7 mirrors 000462 1787865479.543032 mapread 0x024422 thru 0x028fd5 (0x4bb4 bytes) 000463 1787865479.567354 trunc from 0x03ce9d to 0x02be5c (0x11040 bytes) 000464 1787865479.660985 mapwrite 0x017499 thru 0x0248a0 (0xd408 bytes) 000466 1787865479.756995 read 0x017b84 thru 0x025923 (0xdda0 bytes) 000468 1787865479.764099 MIRROR_RESYNC 7 mirrors 000470 1787865481.246811 mapread 0x00dc3b thru 0x01821a (0xa5e0 bytes) 000473 1787865481.333344 mapread 0x028dae thru 0x02a775 (0x19c8 bytes) 000474 1787865481.353925 write 0x030921 thru 0x039ea7 (0x9587 bytes) 000475 1787865481.541773 mapwrite 0x027d05 thru 0x030e39 (0x9135 bytes) 000476 1787865481.643586 mapread 0x00fe6a thru 0x0193db (0x9572 bytes) 000479 1787865481.678382 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39ea8): Mirror 6: 0xf7b06364 (stale) Mirror 7: 0xf7b06364 (stale) Mirror 8: 0xf7b06364 (stale) Mirror 10: 0xf7b06364 (stale) Mirror 12: 0xf7b06364 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ea8): Mirror 6: 0xf7b06364 (stale) Mirror 7: 0xf7b06364 (stale) Mirror 8: 0xf7b06364 (stale) Mirror 10: 0xf7b06364 (stale) Mirror 12: 0xf7b06364 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ea8: skipped 000480 1787865482.321065 mapwrite 0x012a3e thru 0x017fa4 (0x5567 bytes) 000481 1787865482.436226 mapread 0x0240a3 thru 0x024f2f (0xe8d bytes) 000482 1787865482.448162 write 0x026904 thru 0x027f7c (0x1679 bytes) 000483 1787865482.449160 mapwrite 0x00e808 thru 0x01884d (0xa046 bytes) 000484 1787865482.522669 mapwrite 0x01e5e6 thru 0x023dbe (0x57d9 bytes) 000485 1787865482.589045 mapwrite 0x0143b1 thru 0x014cb2 (0x902 bytes) 000486 1787865482.767894 MIRROR_RESYNC 6 mirrors 000489 1787865483.926017 mapwrite 0x00a6fe thru 0x013cf7 (0x95fa bytes) 000490 1787865484.176128 trunc from 0x039ea8 to 0x02df9e (0xbf09 bytes) 000492 1787865484.243599 MIRROR_RESYNC 6 mirrors 000493 1787865485.489489 mapwrite 0x0308a9 thru 0x03fe59 (0xf5b1 bytes) 000494 1787865485.681340 trunc from 0x03fe5a to 0x024539 (0x1b920 bytes) 000495 1787865485.722752 write 0x01c537 thru 0x023a42 (0x750c bytes) 000496 1787865485.746589 trunc from 0x02453a to 0x02da11 (0x94d8 bytes) 000497 1787865485.774692 mapread 0x00147d thru 0x008966 (0x74ea bytes) 000498 1787865485.784073 trunc from 0x02da12 to 0x02885d (0x51b4 bytes) 000499 1787865485.805366 MIRROR_RESYNC 6 mirrors 000500 1787865487.823958 read 0x005243 thru 0x00c476 (0x7234 bytes) 000501 1787865487.834450 trunc from 0x02885e to 0x0190e7 (0xf776 bytes) 000502 1787865487.926381 trunc from 0x0190e8 to 0x0364e4 (0x1d3fd bytes) 000503 1787865487.957751 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x364e5): Mirror 6: 0x96d9d49 (stale) Mirror 7: 0x96d9d49 (stale) Mirror 8: 0x96d9d49 (stale) Mirror 10: 0x96d9d49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x364e5): Mirror 6: 0x96d9d49 (stale) Mirror 7: 0x96d9d49 (stale) Mirror 8: 0x96d9d49 (stale) Mirror 10: 0x96d9d49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x364e5: skipped 000506 1787865488.381794 mapwrite 0x01dd40 thru 0x022e2d (0x50ee bytes) 000507 1787865488.463869 write 0x03ff81 thru 0x03ffff (0x7f bytes) 000508 1787865488.470830 MIRROR_RESYNC 5 mirrors 000510 1787865490.120396 write 0x027541 thru 0x0289e1 (0x14a1 bytes) 000511 1787865490.233478 MIRROR_RESYNC 5 mirrors 000516 1787865492.094822 MIRROR_RESYNC 5 mirrors 000517 1787865492.606028 mapwrite 0x0335a0 thru 0x0395ec (0x604d bytes) 000520 1787865492.748333 mapwrite 0x01e1db thru 0x02d93c (0xf762 bytes) 000524 1787865492.831531 trunc from 0x040000 to 0x01bebe (0x24141 bytes) 000525 1787865492.872629 mapwrite 0x02333d thru 0x032926 (0xf5ea bytes) 000526 1787865492.983233 write 0x023da4 thru 0x03353e (0xf79b bytes) 000527 1787865492.984802 mapwrite 0x008836 thru 0x00b610 (0x2ddb bytes) 000530 1787865493.052318 mapwrite 0x005491 thru 0x010aa2 (0xb612 bytes) 000531 1787865493.106181 write 0x02011f thru 0x02ef09 (0xedeb bytes) 000532 1787865493.112333 MIRROR_RESYNC 5 mirrors 000535 1787865494.745760 read 0x00ecd1 thru 0x01dca3 (0xefd3 bytes) 000536 1787865494.768323 MIRROR_RESYNC 5 mirrors 000538 1787865495.455935 read 0x020c85 thru 0x025a53 (0x4dcf bytes) 000540 1787865495.470808 read 0x006a9e thru 0x01213a (0xb69d bytes) 000541 1787865495.485281 read 0x0136a1 thru 0x01f215 (0xbb75 bytes) 000543 1787865495.487049 mapwrite 0x037ae5 thru 0x03ffff (0x851b bytes) 000544 1787865495.737959 mapwrite 0x028656 thru 0x0310f7 (0x8aa2 bytes) 000548 1787865495.813564 MIRROR_RESYNC 5 mirrors 000549 1787865497.832944 mapread 0x01bb55 thru 0x025798 (0x9c44 bytes) 000550 1787865497.868244 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000551 1787865498.507196 mapwrite 0x002bb9 thru 0x00b45e (0x88a6 bytes) 000552 1787865498.673760 MIRROR_RESYNC 4 mirrors 000553 1787865500.941843 write 0x001935 thru 0x00bfc2 (0xa68e bytes) 000555 1787865501.091875 trunc from 0x040000 to 0x02244f (0x1dbb0 bytes) 000557 1787865501.130912 read 0x020fd4 thru 0x02244f (0x147c bytes) 000558 1787865501.148035 trunc from 0x022450 to 0x031f31 (0xfae2 bytes) 000560 1787865501.176428 read 0x0281c3 thru 0x031f31 (0x9d6f bytes) 000561 1787865501.191280 read 0x02862d thru 0x02eafa (0x64ce bytes) 000564 1787865501.216225 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31f32): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f32): Mirror 6: 0x7bd1c2b (stale) Mirror 8: 0x7bd1c2b (stale) Mirror 10: 0x7bd1c2b (stale) CRC-32 checksum value for chunk [0, 0x31f32): Mirror 1: 0xfb2c7fee Mirror 11: 0xfb2c7fee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f32: skipped 000566 1787865502.377362 trunc from 0x031f32 to 0x01d3af (0x14b82 bytes) 000569 1787865502.488564 write 0x036772 thru 0x03ffff (0x988e bytes) 000570 1787865502.489976 read 0x02a21a thru 0x02fbea (0x59d1 bytes) 000571 1787865502.498591 mapread 0x01da11 thru 0x0220dd (0x46cd bytes) 000572 1787865502.520205 write 0x012c68 thru 0x019b20 (0x6eb9 bytes) 000574 1787865502.527964 mapwrite 0x025dc0 thru 0x02b0aa (0x52eb bytes) 000575 1787865502.612372 read 0x03d95b thru 0x03ffff (0x26a5 bytes) 000578 1787865502.616378 trunc from 0x040000 to 0x02812b (0x17ed4 bytes) 000579 1787865502.639887 mapwrite 0x004a83 thru 0x009136 (0x46b4 bytes) 000580 1787865502.697997 read 0x01d896 thru 0x02087c (0x2fe7 bytes) 000581 1787865502.703643 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2812c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2812c): Mirror 6: 0xfa7f81eb (stale) Mirror 8: 0xfa7f81eb (stale) Mirror 10: 0xfa7f81eb (stale) Mirror 11: 0xa52c0ebd (stale) CRC-32 checksum value for chunk [0, 0x2812c): Mirror 1: 0xf5bed31e Mirror 12: 0xf5bed31e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2812c: skipped 000583 1787865503.921283 mapwrite 0x008287 thru 0x00846f (0x1e9 bytes) 000585 1787865504.075870 write 0x030fe2 thru 0x033f77 (0x2f96 bytes) 000586 1787865504.077840 trunc from 0x033f78 to 0x0373ee (0x3477 bytes) 000587 1787865504.100866 read 0x004834 thru 0x01040d (0xbbda bytes) 000588 1787865504.112022 trunc from 0x0373ef to 0x02898e (0xea60 bytes) 000589 1787865504.129711 write 0x0009d1 thru 0x006e72 (0x64a2 bytes) 000590 1787865504.136638 trunc from 0x02898f to 0x00b5ea (0x1d3a4 bytes) 000591 1787865504.165818 read 0x00a67e thru 0x00b5ea (0xf6d bytes) 000595 1787865504.176452 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xb5eb): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb5eb): Mirror 6: 0xb0681815 (stale) Mirror 8: 0xb0681815 (stale) Mirror 10: 0xb0681815 (stale) Mirror 11: 0x27a3c9fe (stale) Mirror 12: 0x9406a7bc (stale) CRC-32 checksum value for chunk [0, 0xb5eb): Mirror 1: 0x714c270 Mirror 13: 0x714c270 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb5eb: skipped 000597 1787865505.601869 read 0x004f52 thru 0x00b5ea (0x6699 bytes) 000599 1787865505.631967 write 0x01f416 thru 0x02c99e (0xd589 bytes) 000601 1787865505.713509 mapread 0x00b014 thru 0x00b280 (0x26d bytes) 000602 1787865505.723872 mapread 0x011643 thru 0x0150e1 (0x3a9f bytes) 000603 1787865505.742659 read 0x007f5c thru 0x00e355 (0x63fa bytes) 000605 1787865505.762092 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c99f): Mirror 8: 0xb18ef73b (stale) Mirror 10: 0xb18ef73b (stale) Mirror 11: 0x53622fac (stale) Mirror 12: 0xf49ffdf4 (stale) Mirror 13: 0x7582bd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c99f): Mirror 8: 0xb18ef73b (stale) Mirror 10: 0xb18ef73b (stale) Mirror 11: 0x53622fac (stale) Mirror 12: 0xf49ffdf4 (stale) Mirror 13: 0x7582bd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c99f: skipped 000609 1787865506.237470 mapwrite 0x03d789 thru 0x03ffff (0x2877 bytes) 000610 1787865506.351233 trunc from 0x040000 to 0x009566 (0x36a99 bytes) 000611 1787865506.382585 mapread 0x005c13 thru 0x0076eb (0x1ad9 bytes) 000613 1787865506.390505 mapread 0x007dae thru 0x009566 (0x17b9 bytes) 000614 1787865506.395281 MIRROR_RESYNC 6 mirrors 000616 1787865507.752768 mapread 0x00217a thru 0x00361b (0x14a2 bytes) 000617 1787865507.765256 write 0x01aea9 thru 0x026f8d (0xc0e5 bytes) 000618 1787865507.868197 read 0x00ef30 thru 0x0132ac (0x437d bytes) 000619 1787865507.896543 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26f8e): Mirror 8: 0x16fe6881 (stale) Mirror 10: 0x16fe6881 (stale) Mirror 11: 0x16fe6881 (stale) Mirror 13: 0x16fe6881 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26f8e): Mirror 8: 0x16fe6881 (stale) Mirror 10: 0x16fe6881 (stale) Mirror 11: 0x16fe6881 (stale) Mirror 13: 0x16fe6881 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26f8e: skipped 000620 1787865508.426477 mapread 0x025b1d thru 0x02630a (0x7ee bytes) 000621 1787865508.438158 read 0x0016b4 thru 0x006385 (0x4cd2 bytes) 000625 1787865508.450306 trunc from 0x026f8e to 0x02d203 (0x6276 bytes) 000627 1787865508.479806 MIRROR_RESYNC 5 mirrors 000630 1787865510.511273 MIRROR_EXTEND to 6 mirrors 000631 1787865511.932198 mapwrite 0x00776c thru 0x0077cd (0x62 bytes) 000634 1787865512.171125 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d204): Mirror 8: 0xf909370a (stale) Mirror 11: 0xf909370a (stale) Mirror 13: 0xf909370a (stale) Mirror 14: 0xf909370a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d204): Mirror 8: 0xf909370a (stale) Mirror 11: 0xf909370a (stale) Mirror 13: 0xf909370a (stale) Mirror 14: 0xf909370a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d204: skipped 000635 1787865512.701276 read 0x0092f5 thru 0x01240f (0x911b bytes) 000637 1787865512.737066 mapwrite 0x01f818 thru 0x020fab (0x1794 bytes) 000639 1787865512.974318 write 0x027e2b thru 0x035521 (0xd6f7 bytes) 000640 1787865512.992929 mapread 0x02b567 thru 0x0349d1 (0x946b bytes) 000641 1787865513.001177 mapwrite 0x01ddd2 thru 0x01f66d (0x189c bytes) 000642 1787865513.138703 read 0x02a7aa thru 0x03499d (0xa1f4 bytes) truncating to largest ever: 0x3efd6 000643 1787865513.139333 trunc from 0x035522 to 0x03efd5 (0x9ab4 bytes) 000644 1787865513.196873 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3efd6): Mirror 8: 0xee57bfe5 (stale) Mirror 13: 0xee57bfe5 (stale) Mirror 14: 0xee57bfe5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3efd6): Mirror 8: 0xee57bfe5 (stale) Mirror 13: 0xee57bfe5 (stale) Mirror 14: 0xee57bfe5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3efd6: skipped 000645 1787865514.079246 mapwrite 0x00a473 thru 0x00e266 (0x3df4 bytes) 000647 1787865514.190545 trunc from 0x03efd6 to 0x038b74 (0x6461 bytes) 000648 1787865514.214848 write 0x003081 thru 0x00ae33 (0x7db3 bytes) 000649 1787865514.221777 write 0x003db1 thru 0x00db3f (0x9d8f bytes) 000650 1787865514.226150 MIRROR_RESYNC 4 mirrors 000652 1787865515.780274 mapread 0x011f09 thru 0x018658 (0x6750 bytes) 000653 1787865515.805805 write 0x0117e4 thru 0x01ba22 (0xa23f bytes) 000654 1787865515.912740 mapwrite 0x036550 thru 0x038811 (0x22c2 bytes) 000657 1787865515.972276 mapwrite 0x02a304 thru 0x02ac00 (0x8fd bytes) 000660 1787865516.018683 trunc from 0x038b75 to 0x020193 (0x189e1 bytes) 000661 1787865516.052839 trunc from 0x020194 to 0x029b8f (0x99fc bytes) 000663 1787865516.075536 write 0x00b2f3 thru 0x0127f4 (0x7502 bytes) 000665 1787865516.095172 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29b90): Mirror 13: 0x461cb26e (stale) Mirror 14: 0x461cb26e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b90): Mirror 13: 0x461cb26e (stale) Mirror 14: 0x461cb26e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b90: skipped 000667 1787865516.518762 mapwrite 0x00b62a thru 0x019030 (0xda07 bytes) 000669 1787865516.658891 mapwrite 0x02765e thru 0x030a45 (0x93e8 bytes) 000670 1787865516.774822 mapread 0x02f12c thru 0x030a45 (0x191a bytes) 000671 1787865516.785902 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30a46): Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30a46): Mirror 13: 0xed644776 (stale) Mirror 14: 0xed644776 (stale) CRC-32 checksum value for chunk [0, 0x30a46): Mirror 1: 0xd0f338d4 Mirror 15: 0xd0f338d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30a46: skipped 000674 1787865519.052846 MIRROR_RESYNC 4 mirrors 000675 1787865520.806360 write 0x004911 thru 0x012a28 (0xe118 bytes) 000676 1787865520.940511 write 0x03585a thru 0x037371 (0x1b18 bytes) 000677 1787865520.941770 mapread 0x032eb4 thru 0x037371 (0x44be bytes) 000679 1787865520.992945 read 0x00cd12 thru 0x016b21 (0x9e10 bytes) 000680 1787865521.024033 read 0x01254e thru 0x015bea (0x369d bytes) 000681 1787865521.026247 mapread 0x01743a thru 0x01b5a9 (0x4170 bytes) 000685 1787865521.049192 mapread 0x0099bc thru 0x01991e (0xff63 bytes) 000686 1787865521.071650 MIRROR_RESYNC 4 mirrors 000687 1787865523.629151 read 0x0209a6 thru 0x02e575 (0xdbd0 bytes) 000688 1787865523.642589 mapwrite 0x024e06 thru 0x0252ce (0x4c9 bytes) 000692 1787865523.882593 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37372): Mirror 14: 0x4d35f807 (stale) Mirror 15: 0x4d35f807 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37372): Mirror 14: 0x4d35f807 (stale) Mirror 15: 0x4d35f807 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37372: skipped 000695 1787865524.474481 write 0x025507 thru 0x02c33a (0x6e34 bytes) 000697 1787865524.490527 write 0x0158ba thru 0x018447 (0x2b8e bytes) 000698 1787865524.507187 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37372): Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37372): Mirror 14: 0x4d35f807 (stale) Mirror 15: 0x4d35f807 (stale) CRC-32 checksum value for chunk [0, 0x37372): Mirror 1: 0x41d82a1e Mirror 16: 0x41d82a1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37372: skipped 000700 1787865526.683337 read 0x01cb07 thru 0x029c88 (0xd182 bytes) 000703 1787865526.720558 mapwrite 0x0274cf thru 0x030bbb (0x96ed bytes) 000704 1787865527.286159 write 0x019f76 thru 0x0249c9 (0xaa54 bytes) 000706 1787865527.324281 trunc from 0x037372 to 0x00c3a6 (0x2afcb bytes) 000707 1787865527.371016 mapread 0x00bee5 thru 0x00c3a6 (0x4c2 bytes) 000709 1787865527.385652 write 0x0011db thru 0x004b1c (0x3942 bytes) 000710 1787865527.400628 write 0x02adb9 thru 0x02b20b (0x453 bytes) 000712 1787865527.424077 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2b20c): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b20c): Mirror 14: 0x67eb91cb (stale) Mirror 15: 0x67eb91cb (stale) Mirror 16: 0xb27e6133 (stale) CRC-32 checksum value for chunk [0, 0x2b20c): Mirror 1: 0x3cd0fe46 Mirror 17: 0x3cd0fe46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b20c: skipped 000714 1787865528.821017 write 0x02a6f0 thru 0x039d5c (0xf66d bytes) 000715 1787865528.871338 read 0x00c2ac thru 0x00c73b (0x490 bytes) 000716 1787865528.876080 trunc from 0x039d5d to 0x023c3f (0x1611d bytes) 000719 1787865528.907785 mapread 0x0025a1 thru 0x008957 (0x63b7 bytes) 000720 1787865528.932702 mapwrite 0x02444c thru 0x030e20 (0xc9d5 bytes) 000723 1787865528.987132 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30e21): Mirror 14: 0xc65d5af (stale) Mirror 15: 0xc65d5af (stale) Mirror 17: 0xbcd8be53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e21): Mirror 14: 0xc65d5af (stale) Mirror 15: 0xc65d5af (stale) Mirror 17: 0xbcd8be53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e21: skipped 000724 1787865529.484023 mapwrite 0x00b74c thru 0x013ecf (0x8784 bytes) 000726 1787865529.577576 write 0x020fe9 thru 0x02f85b (0xe873 bytes) 000729 1787865529.603157 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30e21): Mirror 15: 0xc65d5af (stale) Mirror 17: 0xbcd8be53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e21): Mirror 15: 0xc65d5af (stale) Mirror 17: 0xbcd8be53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e21: skipped 000732 1787865530.101703 MIRROR_RESYNC 3 mirrors 000739 1787865531.998340 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000740 1787865532.595268 trunc from 0x030e21 to 0x03c51b (0xb6fb bytes) 000743 1787865532.783344 write 0x03101d thru 0x03b57a (0xa55e bytes) 000744 1787865532.802123 read 0x017feb thru 0x019c8d (0x1ca3 bytes) 000745 1787865532.810672 trunc from 0x03c51c to 0x026230 (0x162eb bytes) 000746 1787865532.852765 mapread 0x025a80 thru 0x026230 (0x7b1 bytes) 000747 1787865532.888666 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000750 1787865533.146318 mapread 0x0096e2 thru 0x00ef32 (0x5851 bytes) 000752 1787865533.193570 read 0x0226b6 thru 0x026230 (0x3b7b bytes) 000754 1787865533.227380 MIRROR_EXTEND to 2 mirrors 000757 1787865534.381816 read 0x017c0f thru 0x01f176 (0x7568 bytes) 000758 1787865534.395255 mapwrite 0x00b4de thru 0x01942f (0xdf52 bytes) 000760 1787865534.669641 read 0x003618 thru 0x00db86 (0xa56f bytes) 000761 1787865534.682654 mapwrite 0x02aa7b thru 0x036671 (0xbbf7 bytes) 000762 1787865534.804711 write 0x01a1c6 thru 0x024d4d (0xab88 bytes) 000763 1787865534.824170 MIRROR_RESYNC 2 mirrors 000764 1787865536.321120 trunc from 0x036672 to 0x003c7a (0x329f7 bytes) 000765 1787865536.525699 mapwrite 0x02e2ec thru 0x03d82c (0xf541 bytes) 000766 1787865536.752617 trunc from 0x03d82d to 0x01286d (0x2afbf bytes) 000767 1787865536.796117 mapwrite 0x01c37b thru 0x022d23 (0x69a9 bytes) 000780 1787865536.904209 mapwrite 0x01c274 thru 0x025f6c (0x9cf9 bytes) 000781 1787865537.032234 trunc from 0x025f6d to 0x0276cc (0x1760 bytes) 000782 1787865537.060161 mapread 0x01127e thru 0x012e01 (0x1b84 bytes) 000785 1787865537.104585 trunc from 0x0276cd to 0x037ecc (0x10800 bytes) 000786 1787865537.134973 mapwrite 0x000c13 thru 0x00b0fd (0xa4eb bytes) 000787 1787865537.303823 write 0x01e7f0 thru 0x028c5f (0xa470 bytes) 000788 1787865537.329018 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000789 1787865537.550276 mapread 0x035e76 thru 0x037ecc (0x2057 bytes) 000790 1787865537.573886 trunc from 0x037ecd to 0x0030f2 (0x34dda bytes) 000791 1787865537.597942 MIRROR_EXTEND to 2 mirrors 000792 1787865538.887993 MIRROR_RESYNC 2 mirrors 000794 1787865539.309957 mapwrite 0x03e042 thru 0x03ffff (0x1fbe bytes) 000797 1787865539.645213 trunc from 0x040000 to 0x02d420 (0x12bdf bytes) 000801 1787865539.692236 mapread 0x018c74 thru 0x0239ba (0xad47 bytes) 000802 1787865539.730828 mapread 0x026eed thru 0x02d420 (0x6534 bytes) 000803 1787865539.748083 read 0x00cb26 thru 0x01288b (0x5d66 bytes) 000805 1787865539.766124 mapread 0x00108a thru 0x00964a (0x85c1 bytes) 000806 1787865539.787274 mapwrite 0x026ea2 thru 0x031ff9 (0xb158 bytes) 000809 1787865539.970002 trunc from 0x031ffa to 0x01fb4b (0x124ae bytes) 000810 1787865539.995743 mapwrite 0x035015 thru 0x03e1a3 (0x918f bytes) 000812 1787865540.128722 read 0x022a68 thru 0x02b137 (0x86d0 bytes) 000813 1787865540.138870 read 0x021aab thru 0x02e2ad (0xc803 bytes) 000815 1787865540.140488 mapwrite 0x022368 thru 0x02c77d (0xa416 bytes) 000816 1787865540.184276 mapread 0x009752 thru 0x00dd4c (0x45fb bytes) 000817 1787865540.185331 mapwrite 0x00e82b thru 0x01475b (0x5f31 bytes) 000818 1787865540.241116 MIRROR_RESYNC 2 mirrors 000820 1787865541.569468 mapread 0x022c32 thru 0x025eb2 (0x3281 bytes) 000823 1787865541.593082 mapwrite 0x00a684 thru 0x015640 (0xafbd bytes) 000825 1787865541.782947 MIRROR_RESYNC 2 mirrors 000826 1787865543.190802 read 0x0101d0 thru 0x013b6b (0x399c bytes) 000829 1787865543.200566 mapread 0x03d8f6 thru 0x03e1a3 (0x8ae bytes) 000830 1787865543.243141 mapread 0x023aa8 thru 0x02f04f (0xb5a8 bytes) 000832 1787865543.297177 mapwrite 0x0206ff thru 0x02e8b3 (0xe1b5 bytes) 000833 1787865543.600012 mapwrite 0x03681b thru 0x036a3b (0x221 bytes) 000834 1787865543.648496 trunc from 0x03e1a4 to 0x00cd5e (0x31445 bytes) 000835 1787865543.677843 trunc from 0x00cd5f to 0x01fb1b (0x12dbd bytes) 000837 1787865543.702928 mapread 0x00a06d thru 0x0161fb (0xc18f bytes) 000840 1787865543.732361 trunc from 0x01fb1c to 0x02928e (0x9773 bytes) 000841 1787865543.750228 mapread 0x011046 thru 0x0116b6 (0x671 bytes) 000842 1787865543.752584 mapwrite 0x005320 thru 0x0086c0 (0x33a1 bytes) 000843 1787865543.804537 trunc from 0x02928f to 0x0130b0 (0x161de bytes) 000845 1787865543.829026 write 0x03526c thru 0x03ffff (0xad94 bytes) 000846 1787865543.835035 mapread 0x00cad3 thru 0x0142f5 (0x7823 bytes) 000849 1787865543.848826 mapread 0x0246d5 thru 0x031842 (0xd16e bytes) 000850 1787865543.868645 write 0x02ba5c thru 0x02deaf (0x2454 bytes) 000851 1787865543.870818 mapread 0x0195b0 thru 0x01cac3 (0x3514 bytes) 000853 1787865543.873659 trunc from 0x040000 to 0x01fe1b (0x201e4 bytes) 000854 1787865543.899294 mapread 0x003dbf thru 0x012652 (0xe894 bytes) 000856 1787865543.902339 trunc from 0x01fe1c to 0x01b207 (0x4c14 bytes) 000857 1787865543.925166 mapwrite 0x00a45b thru 0x01a313 (0xfeb9 bytes) 000859 1787865544.028443 trunc from 0x01b208 to 0x003cd3 (0x17534 bytes) 000860 1787865544.094012 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cd4): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd4): Mirror 2: 0xde884326 (stale) CRC-32 checksum value for chunk [0, 0x3cd4): Mirror 1: 0xde884326 Mirror 3: 0xde884326 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd4: skipped 000863 1787865545.708122 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000867 1787865546.241129 trunc from 0x003cd4 to 0x01f473 (0x1b7a0 bytes) 000871 1787865546.417339 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1f474): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f474): Mirror 3: 0x8f4de3d7 (stale) CRC-32 checksum value for chunk [0, 0x1f474): Mirror 1: 0x8f4de3d7 Mirror 4: 0x8f4de3d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f474: skipped 000873 1787865548.301821 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1f474): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f474): Mirror 3: 0x8f4de3d7 (stale) CRC-32 checksum value for chunk [0, 0x1f474): Mirror 1: 0x8f4de3d7 Mirror 4: 0x8f4de3d7 Mirror 5: 0x8f4de3d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f474: skipped 000874 1787865549.661992 write 0x033100 thru 0x03ffff (0xcf00 bytes) 000875 1787865549.735977 trunc from 0x040000 to 0x018451 (0x27bae bytes) 000877 1787865549.759215 mapread 0x015ee4 thru 0x018451 (0x256e bytes) 000880 1787865549.789135 mapwrite 0x012799 thru 0x01dce2 (0xb54a bytes) 000881 1787865549.969062 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dce3): Mirror 4: 0xd21bcdc3 (stale) Mirror 5: 0xd21bcdc3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dce3): Mirror 4: 0xd21bcdc3 (stale) Mirror 5: 0xd21bcdc3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dce3: skipped 000882 1787865550.295244 trunc from 0x01dce3 to 0x0066b0 (0x17632 bytes) 000884 1787865550.308801 read 0x001cc5 thru 0x0066b0 (0x49ec bytes) 000885 1787865550.315745 mapwrite 0x0195ce thru 0x024b04 (0xb537 bytes) 000888 1787865550.402556 MIRROR_RESYNC 3 mirrors 000889 1787865551.534644 mapread 0x00f6d0 thru 0x01651a (0x6e4b bytes) 000891 1787865551.558860 trunc from 0x024b05 to 0x021777 (0x338d bytes) 000894 1787865551.661308 trunc from 0x021778 to 0x037525 (0x15dae bytes) 000895 1787865551.685921 mapwrite 0x033eb8 thru 0x035fc4 (0x210d bytes) 000896 1787865551.729196 trunc from 0x037526 to 0x03b34a (0x3e25 bytes) 000897 1787865551.752206 mapread 0x038ffd thru 0x03b34a (0x234e bytes) 000899 1787865551.764344 write 0x00b85d thru 0x0151c9 (0x996d bytes) 000900 1787865551.800309 write 0x019e4e thru 0x02231c (0x84cf bytes) 000901 1787865551.821518 mapwrite 0x0385e0 thru 0x03954d (0xf6e bytes) 000903 1787865551.968717 trunc from 0x03b34b to 0x025406 (0x15f44 bytes) 000905 1787865552.020026 trunc from 0x025407 to 0x039358 (0x13f52 bytes) 000906 1787865552.064171 read 0x028ac0 thru 0x02cae7 (0x4028 bytes) 000908 1787865552.070289 mapwrite 0x03f89f thru 0x03ffff (0x761 bytes) 000909 1787865552.146392 mapwrite 0x02d8a3 thru 0x0387b5 (0xaf13 bytes) 000912 1787865552.316183 mapwrite 0x00da24 thru 0x010b16 (0x30f3 bytes) 000913 1787865552.419075 trunc from 0x040000 to 0x00ac3e (0x353c1 bytes) 000914 1787865552.473277 write 0x021239 thru 0x024352 (0x311a bytes) 000917 1787865552.481501 read 0x017c66 thru 0x01f70f (0x7aaa bytes) 000919 1787865552.514989 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24353): Mirror 5: 0x61c08492 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24353): Mirror 5: 0x61c08492 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24353: skipped 000920 1787865552.901036 mapwrite 0x0040f2 thru 0x00fa64 (0xb973 bytes) 000922 1787865552.979211 read 0x0228a7 thru 0x024352 (0x1aac bytes) 000929 1787865553.003939 mapwrite 0x02bc28 thru 0x02d003 (0x13dc bytes) 000930 1787865553.110641 trunc from 0x02d004 to 0x0394c7 (0xc4c4 bytes) 000931 1787865553.156567 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x394c8): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394c8): Mirror 5: 0xd844248c (stale) CRC-32 checksum value for chunk [0, 0x394c8): Mirror 1: 0xc22a1d82 Mirror 6: 0xc22a1d82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394c8: skipped 000933 1787865555.346048 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x394c8): Mirror 5: 0xd844248c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394c8): Mirror 5: 0xd844248c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394c8: skipped 000935 1787865555.840554 trunc from 0x0394c8 to 0x022ec4 (0x16603 bytes) 000937 1787865555.886885 write 0x018eca thru 0x0214ea (0x8621 bytes) 000938 1787865555.902466 trunc from 0x022ec5 to 0x03cc5a (0x19d96 bytes) 000940 1787865555.924517 read 0x01bd8d thru 0x01d4c7 (0x173b bytes) 000942 1787865555.926343 mapread 0x025ec6 thru 0x02792c (0x1a67 bytes) 000943 1787865555.950847 trunc from 0x03cc5b to 0x008f15 (0x33d45 bytes) 000944 1787865555.975259 read 0x00515b thru 0x008f15 (0x3dbb bytes) 000945 1787865555.989814 mapread 0x00553d thru 0x007fb7 (0x2a7b bytes) 000947 1787865555.993257 write 0x03d0db thru 0x03ffff (0x2f25 bytes) 000948 1787865555.995028 mapread 0x03e592 thru 0x03f17e (0xbed bytes) 000949 1787865555.998964 write 0x0044d1 thru 0x00a86a (0x639a bytes) 000950 1787865556.018477 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000951 1787865556.295853 mapwrite 0x026c4d thru 0x033479 (0xc82d bytes) 000957 1787865556.422308 trunc from 0x040000 to 0x038485 (0x7b7a bytes) 000958 1787865556.468174 MIRROR_EXTEND to 2 mirrors 000959 1787865557.509318 trunc from 0x038486 to 0x03cbea (0x4765 bytes) 000963 1787865557.629738 trunc from 0x03cbeb to 0x00d12a (0x2fac0 bytes) 000965 1787865557.668508 trunc from 0x00d12b to 0x0199b0 (0xc886 bytes) 000966 1787865557.696698 mapwrite 0x03efcb thru 0x03ffff (0x1035 bytes) 000968 1787865557.784587 mapread 0x008daf thru 0x017651 (0xe8a3 bytes) 000969 1787865557.816521 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x168:0x0]: Invalid argument (22) 000971 1787865557.984093 write 0x0029fb thru 0x00d50d (0xab13 bytes) 000973 1787865557.997965 mapread 0x002ee4 thru 0x00ade4 (0x7f01 bytes) 000977 1787865558.002165 mapwrite 0x038e3b thru 0x03ab88 (0x1d4e bytes) 000978 1787865558.079383 mapwrite 0x0318d1 thru 0x03f681 (0xddb1 bytes) 000981 1787865558.145317 mapread 0x01d2db thru 0x02a61a (0xd340 bytes) 000987 1787865558.169303 write 0x0386cb thru 0x03ffff (0x7935 bytes) 000989 1787865558.171317 read 0x014d6c thru 0x015fac (0x1241 bytes) 000991 1787865558.178975 mapread 0x001608 thru 0x0069c8 (0x53c1 bytes) 000996 1787865558.191453 mapread 0x035d8e thru 0x03755a (0x17cd bytes) 000998 1787865558.193910 mapwrite 0x01680e thru 0x023801 (0xcff4 bytes) 000999 1787865558.254010 mapread 0x03424c thru 0x03ffff (0xbdb4 bytes) All operations completed A-OK! PASS 70a (156s) == sanity-flr test 71: check mirror extend parent fid ==== 17:19:23 (1787865563) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 17:19:27 (1787865567) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.320171 s, 6.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147693 s, 7.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x283:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x284:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x29f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x285:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 17:19:38 (1787865578) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6424192 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1085024 .. failed Extending file size to 7742208 .. Extending file size to 2248160 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8048000 .. Extending file size to 8857152 .. Extending file size to 4375104 .. Extending file size to 7929440 .. Extending file size to 9182656 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8392672 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8863136 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8983936 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58645 58646 58649 58650 58651 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) 192.168.202.114@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:21:00 (1787865660) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7301024 .. Extending file size to 3790720 .. Extending file size to 4074688 .. Extending file size to 7346368 .. Extending file size to 7496064 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7471488 .. Extending file size to 8037376 .. Extending file size to 8348512 .. Extending file size to 8140416 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8254400 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8564160 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9304320 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8497472 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 60259 60260 60262 60264 60266 failed 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) 192.168.202.114@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:22:19 (1787865739) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201433 s, 10.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.625453 s, 3.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xc59128c5 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:22:44 (1787865764) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153611 s, 13.7 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 17:22:53 (1787865773) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bc:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bd:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2a0:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:23:02 (1787865782) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.253416 s, 8.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:23:10 (1787865790) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.287742 s, 7.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a3:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a4:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:23:20 (1787865800) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.291581 s, 7.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a6:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:23:54 (1787865834) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg214-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103116 s, 10.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:24:51 (1787865891) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg214-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.400633 s, 5.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:25:53 (1787865953) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg214-server mds-ost sync done. Creating new pool test_204f oleg214-server: Pool lustre.test_204f created Adding targets to pool oleg214-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg214-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.493392 s, 4.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg214-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg214-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg214-server: Pool lustre.test_204f destroyed PASS 204f (87s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:27:20 (1787866040) PASS 205a (12s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:27:32 (1787866052) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:27:42 (1787866062) oleg214-server: Pool lustre.test_206 created Adding targets to pool oleg214-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg214-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg214-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg214-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg214-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:28:11 (1787866091) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101179 s, 10.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547493 s, 19.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8fd0fd0d Mirror 3: 0x8fd0fd0d PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:28:26 (1787866106) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:28:30 (1787866110) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:28:34 (1787866114) Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.130787 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1400000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1400000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1400000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12e0000 Read this (4 bytes): BBBB 192.168.202.114@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:29:16 (1787866156) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0275069 s, 298 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:29:24 (1787866164) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575139 s, 18.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d8 lmm_fid: [0x200000404:0x1d8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d8 lmm_fid: [0x200000404:0x1d8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834392 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 17:29:26.000000000 -0400 Modify: 2026-08-27 17:29:26.000000000 -0400 Change: 2026-08-27 17:29:27.000000000 -0400 Birth: 2026-08-27 17:29:26.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:29:33 (1787866173) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:29:37 (1787866177) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.571129 s, 71.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.049929 s, 82.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 17:29 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1da:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 17:29 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:29:46 (1787866186) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:29:54 (1787866194) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1940 sec =========== 17:30:05 (1787866205) === sanity-flr: start cleanup 17:30:07 (1787866207) === === sanity-flr: finish cleanup 17:30:15 (1787866215) ===