-----============= acceptance-small: sanity-flr ============----- Sat May 9 15:28:03 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg301-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:28:11 (1778354891) === oleg301-client.virtnet: executing check_config_client /mnt/lustre oleg301-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg301-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d57050cd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d57050cd000.idle_timeout=debug disable quota as required oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:28:22 (1778354902) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7541] == sanity-flr test 0a: lfs mirror create with -N option == 15:28:23 (1778354903) PASS 0a (4s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:28:27 (1778354907) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:28:28 (1778354908) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:28:29 (1778354909) PASS 0d (6s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:28:35 (1778354915) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:28:36 (1778354916) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 15:28:37 (1778354917) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d57050cd000.stats=clear osc.lustre-OST0001-osc-ffff8d57050cd000.stats=clear PASS 0g (3s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:28:40 (1778354920) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (3s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:28:43 (1778354923) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:28:45 (1778354925) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:28:46 (1778354926) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:28:49 (1778354929) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 15:28:52 (1778354932) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:28:54 (1778354934) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:28:55 (1778354935) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:28:57 (1778354937) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:29:01 (1778354941) PASS 7 (2s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:29:03 (1778354943) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.265885 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029311 s, 35.8 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:29:06 (1778354946) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0484015 s, 108 MB/s PASS 22 (2s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:29:08 (1778354948) Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d57050cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d57050cd000.ost_server_uuid in DISCONN state after 2 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d57050cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d57050cd000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (27s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:29:35 (1778354975) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0802056 s, 52.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d57050cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d57050cd000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (11s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:29:46 (1778354986) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d57050cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d57050cd000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d57050cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d57050cd000.ost_server_uuid in DISCONN state after 1 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (20s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:30:06 (1778355006) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.03927 s, 50.4 MB/s -rw-r--r-- 1 root root 52428800 May 9 15:30 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.227s user 0m0.000s sys 0m0.132s Stopping /mnt/lustre-ost1 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d57050cd000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d57050cd000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... real 0m0.279s user 0m0.000s sys 0m0.159s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg301-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg301-server oleg301-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d57050cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d57050cd000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.288s user 0m0.001s sys 0m0.161s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg301-server: oleg301-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg301-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d57050cd000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d57050cd000.ost_server_uuid in FULL state after 0 sec PASS 33b (27s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:30:33 (1778355033) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:30:34 (1778355034) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:30:35 (1778355035) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 15:30:36 (1778355036) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 15:30:39 (1778355039) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.138883 s, 90.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (3s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:30:42 (1778355042) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128657 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00936392 s, 112 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:30:45 (1778355045) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:30:48 (1778355048) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000577024 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000517837 s, 2.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000711562 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000606542 s, 1.7 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 15:30:51 (1778355051) create test files with size 15744897 .. 176fce99a3be6144a38e0686eb967d61 - b0824c5722325a5d62aa24ac6ea6a732 - bccd4623da10635564937c55e98f266b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre PASS 37 (6s) == sanity-flr test 38: resync ============================ 15:30:57 (1778355057) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:31:01 (1778355061) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:31:03 (1778355063) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401189 s, 26.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x47:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989437 s, 106 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x48:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x49:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (2s) == sanity-flr test 41: lfs mirror resync check =========== 15:31:05 (1778355065) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0395714 s, 53.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.129459 s, 32.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0502879 s, 41.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (5s) == sanity-flr test 42: lfs mirror verify ================= 15:31:10 (1778355070) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 15:31:11 (1778355071) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:31:12 (1778355072) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00953123 s, 110 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (2s) == sanity-flr test 44a: lfs mirror split check =========== 15:31:14 (1778355074) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:31:15 (1778355075) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x50:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x54:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x54:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:31:19 (1778355079) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.111882 s, 93.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (20s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:31:39 (1778355099) llite.lustre-ffff8d571088e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0861129 s, 122 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg301-client.virtnet /mnt/lustre (opts:) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff8d57030d2800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:31:43 (1778355103) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (13s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:31:56 (1778355116) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:32:06 (1778355126) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 15:32:07 (1778355127) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0736078 s, 42.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0493227 s, 63.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1d0130fa (stale) Mirror 2: 0x1d0130fa (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: bf317451a0fe62698f6f585870bc739e - mirror 1 checksum: 32341c25830bc947fddac8361344406c - mirror 2 checksum: bf317451a0fe62698f6f585870bc739e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat May 9 15:32:08 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Sat May 9 15:32:08 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: bf317451a0fe62698f6f585870bc739e - mirror 1 checksum: 32341c25830bc947fddac8361344406c - mirror 2 checksum: 32341c25830bc947fddac8361344406c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat May 9 15:32:08 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x66:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Sat May 9 15:32:08 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x67:0x0] } PASS 48 (3s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:32:11 (1778355131) Failing mds1 on oleg301-server Stopping /mnt/lustre-mds1 (opts:) on oleg301-server 15:32:13 (1778355133) shut down facet: mds1 facet_host: oleg301-server facet_failover_host: oleg301-server Failover mds1 to oleg301-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 15:32:27 (1778355147) targets are mounted 15:32:27 (1778355147) facet_failover done oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013952 s, 75.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x68:0x0] } PASS 50A (22s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:32:33 (1778355153) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:32:34 (1778355154) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:32:35 (1778355155) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00252148 s, 6.5 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:32:37 (1778355157) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:32:38 (1778355158) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (1s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:32:39 (1778355159) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (6s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:32:45 (1778355165) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:33:04 (1778355184) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (18s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:33:22 (1778355202) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:33:40 (1778355220) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 15:33:42 (1778355222) mirror create pid 52779 mirror split pid 52780 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52779 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52780 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 15:34:45 (1778355285) Seed set to 5286 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x3e41f 000005 1778355286.097524 trunc from 00000000 to 0x03e41e (0x3e41f bytes) 000007 1778355286.104427 write 0x020035 thru 0x02d608 (0xd5d4 bytes) 000008 1778355286.112296 MIRROR_EXTEND to 2 mirrors 000009 1778355286.389446 mapread 0x028335 thru 0x02bd27 (0x39f3 bytes) 000010 1778355286.398282 write 0x027b38 thru 0x029644 (0x1b0d bytes) 000011 1778355286.425987 write 0x033145 thru 0x03df9c (0xae58 bytes) 000012 1778355286.433364 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e41f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e41f): Mirror 2: 0x7e628860 (stale) CRC-32 checksum value for chunk [0, 0x3e41f): Mirror 1: 0xfdb90879 Mirror 3: 0xfdb90879 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e41f: skipped 000014 1778355286.834232 mapwrite 0x03c007 thru 0x03ffff (0x3ff9 bytes) 000016 1778355286.887673 trunc from 0x040000 to 0x035023 (0xafdc bytes) 000017 1778355286.895077 mapwrite 0x0219d7 thru 0x02b98e (0x9fb8 bytes) 000022 1778355286.918910 mapread 0x002ab2 thru 0x00d6bc (0xac0b bytes) 000023 1778355286.922537 trunc from 0x035024 to 0x037aa9 (0x2a86 bytes) 000025 1778355286.928886 read 0x00015f thru 0x00d625 (0xd4c7 bytes) 000026 1778355286.930988 mapwrite 0x02e5ee thru 0x033a17 (0x542a bytes) 000027 1778355286.952928 mapwrite 0x029f0e thru 0x02c2d8 (0x23cb bytes) 000031 1778355286.977114 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37aaa): Mirror 3: 0xe1b65460 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37aaa): Mirror 3: 0xe1b65460 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37aaa: skipped 000033 1778355287.112547 mapread 0x00ce50 thru 0x012b4a (0x5cfb bytes) 000035 1778355287.127832 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000039 1778355287.198936 write 0x0191e5 thru 0x026ab8 (0xd8d4 bytes) 000040 1778355287.206136 mapwrite 0x03672a thru 0x03ffff (0x98d6 bytes) 000041 1778355287.249973 read 0x020b2e thru 0x02ab19 (0x9fec bytes) 000044 1778355287.255064 write 0x011b41 thru 0x01b1fb (0x96bb bytes) 000045 1778355287.259285 read 0x0299f5 thru 0x02d43a (0x3a46 bytes) 000050 1778355287.265086 write 0x02e0a9 thru 0x02f7e9 (0x1741 bytes) 000052 1778355287.266166 mapwrite 0x011af2 thru 0x0129fc (0xf0b bytes) 000056 1778355287.283020 mapwrite 0x03d84d thru 0x03ffff (0x27b3 bytes) 000057 1778355287.298269 mapwrite 0x017213 thru 0x025dda (0xebc8 bytes) 000059 1778355287.319606 read 0x02ab21 thru 0x0397d7 (0xecb7 bytes) 000060 1778355287.320269 write 0x00158c thru 0x002447 (0xebc bytes) 000062 1778355287.321518 read 0x02a1a2 thru 0x02e44c (0x42ab bytes) 000064 1778355287.322641 read 0x01013b thru 0x0149eb (0x48b1 bytes) 000070 1778355287.326210 mapread 0x028a06 thru 0x033218 (0xa813 bytes) 000073 1778355287.328520 read 0x0392b6 thru 0x03ffff (0x6d4a bytes) 000075 1778355287.329592 write 0x00b93f thru 0x00e8d3 (0x2f95 bytes) 000077 1778355287.331062 mapread 0x00d528 thru 0x00fc69 (0x2742 bytes) 000079 1778355287.332257 trunc from 0x040000 to 0x012d1d (0x2d2e2 bytes) 000083 1778355287.341933 mapread 0x0057bb thru 0x00cd6c (0x75b2 bytes) 000084 1778355287.345378 MIRROR_EXTEND to 2 mirrors 000086 1778355287.607970 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000089 1778355287.672362 read 0x0074e0 thru 0x008556 (0x1077 bytes) 000090 1778355287.675993 write 0x006993 thru 0x012f84 (0xc5f2 bytes) 000092 1778355287.687335 write 0x015099 thru 0x02179b (0xc703 bytes) 000093 1778355287.688973 mapwrite 0x0146f3 thru 0x017f22 (0x3830 bytes) 000094 1778355287.715207 mapread 0x00cc58 thru 0x01ad90 (0xe139 bytes) 000097 1778355287.717541 trunc from 0x02179c to 0x032d09 (0x1156e bytes) 000098 1778355287.733375 write 0x029c1a thru 0x033c4e (0xa035 bytes) 000099 1778355287.738498 mapread 0x00bcd5 thru 0x017859 (0xbb85 bytes) 000102 1778355287.741425 read 0x00d927 thru 0x01c78a (0xee64 bytes) 000104 1778355287.742999 mapwrite 0x021821 thru 0x023dad (0x258d bytes) 000106 1778355287.765392 mapread 0x01cbe0 thru 0x022035 (0x5456 bytes) 000108 1778355287.767879 trunc from 0x033c4f to 0x021056 (0x12bf8 bytes) 000109 1778355287.776159 mapread 0x014e52 thru 0x021056 (0xc205 bytes) 000114 1778355287.779788 write 0x006581 thru 0x013675 (0xd0f5 bytes) 000116 1778355287.781764 read 0x01036e thru 0x01ab07 (0xa79a bytes) 000117 1778355287.782408 write 0x0075a5 thru 0x013932 (0xc38e bytes) 000118 1778355287.783746 read 0x00914c thru 0x00929a (0x14f bytes) 000119 1778355287.784374 mapwrite 0x01f70f thru 0x0212c4 (0x1bb6 bytes) 000120 1778355287.810575 read 0x01384f thru 0x017d2c (0x44de bytes) 000122 1778355287.812182 read 0x01e65d thru 0x0212c4 (0x2c68 bytes) 000124 1778355287.813786 mapwrite 0x0206e4 thru 0x02b166 (0xaa83 bytes) 000125 1778355287.848959 read 0x00f745 thru 0x012d7f (0x363b bytes) 000128 1778355287.851390 write 0x027620 thru 0x03373b (0xc11c bytes) 000131 1778355287.855256 trunc from 0x03373c to 0x010c40 (0x22afb bytes) 000132 1778355287.864887 mapwrite 0x03c8e1 thru 0x03ffff (0x371f bytes) 000133 1778355287.899542 trunc from 0x040000 to 0x001137 (0x3eec8 bytes) 000136 1778355287.908930 trunc from 0x001138 to 0x03db8d (0x3ca56 bytes) 000139 1778355287.919182 mapread 0x030ffc thru 0x03db8d (0xcb92 bytes) 000142 1778355287.958292 mapwrite 0x02b4b7 thru 0x02e9c9 (0x3513 bytes) 000143 1778355287.995172 trunc from 0x03db8e to 0x010b1a (0x2d073 bytes) 000145 1778355288.005879 mapwrite 0x0287a1 thru 0x02ec32 (0x6492 bytes) 000146 1778355288.060199 write 0x01a849 thru 0x022717 (0x7ecf bytes) 000147 1778355288.066888 mapwrite 0x02247e thru 0x0276d3 (0x5256 bytes) 000148 1778355288.108301 read 0x023fd8 thru 0x02ec32 (0xac5b bytes) 000151 1778355288.110879 trunc from 0x02ec33 to 0x012458 (0x1c7da bytes) 000152 1778355288.119863 write 0x00d8b0 thru 0x00f992 (0x20e3 bytes) 000155 1778355288.126331 trunc from 0x012459 to 0x010fb2 (0x14a6 bytes) 000158 1778355288.137937 MIRROR_EXTEND to 2 mirrors 000159 1778355288.426048 trunc from 0x010fb3 to 0x013ca4 (0x2cf2 bytes) 000160 1778355288.469949 write 0x014ed9 thru 0x0184c4 (0x35ec bytes) 000161 1778355288.471535 read 0x010572 thru 0x0119b2 (0x1441 bytes) 000162 1778355288.475622 mapwrite 0x02a91d thru 0x034eab (0xa58f bytes) 000163 1778355288.528387 write 0x03e291 thru 0x03ffff (0x1d6f bytes) 000164 1778355288.529622 mapwrite 0x0205f9 thru 0x0228da (0x22e2 bytes) 000166 1778355288.556941 read 0x01801f thru 0x01dec8 (0x5eaa bytes) 000167 1778355288.557892 write 0x000fab thru 0x0077fc (0x6852 bytes) 000168 1778355288.563530 read 0x01c3e9 thru 0x028d20 (0xc938 bytes) 000169 1778355288.564516 mapwrite 0x03775e thru 0x03f237 (0x7ada bytes) 000171 1778355288.586716 mapread 0x021069 thru 0x02400f (0x2fa7 bytes) 000172 1778355288.588052 read 0x009938 thru 0x019872 (0xff3b bytes) 000173 1778355288.588968 read 0x035943 thru 0x03a1aa (0x4868 bytes) 000175 1778355288.590294 write 0x03ed15 thru 0x03ffff (0x12eb bytes) 000178 1778355288.592658 read 0x024f7b thru 0x028da6 (0x3e2c bytes) 000180 1778355288.593933 mapwrite 0x03c33c thru 0x03ffff (0x3cc4 bytes) 000183 1778355288.617361 read 0x031489 thru 0x036195 (0x4d0d bytes) 000187 1778355288.620565 mapread 0x01bbf5 thru 0x020859 (0x4c65 bytes) 000188 1778355288.621939 mapwrite 0x01af78 thru 0x02599c (0xaa25 bytes) 000189 1778355288.648419 write 0x023565 thru 0x03035f (0xcdfb bytes) 000191 1778355288.650518 write 0x000ce6 thru 0x00ed03 (0xe01e bytes) 000193 1778355288.652656 mapread 0x01b324 thru 0x022258 (0x6f35 bytes) 000194 1778355288.653881 mapwrite 0x035e0a thru 0x03ffff (0xa1f6 bytes) 000195 1778355288.677140 read 0x010bff thru 0x011fe0 (0x13e2 bytes) 000196 1778355288.677996 write 0x010c4a thru 0x015f2a (0x52e1 bytes) 000198 1778355288.679669 mapread 0x030edc thru 0x033d05 (0x2e2a bytes) 000200 1778355288.683544 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000201 1778355288.752457 mapwrite 0x013261 thru 0x019376 (0x6116 bytes) 000202 1778355288.787085 mapread 0x00d4af thru 0x00e9b6 (0x1508 bytes) 000204 1778355288.793482 write 0x0213d7 thru 0x02617e (0x4da8 bytes) 000206 1778355288.798010 mapread 0x02f7d7 thru 0x033f50 (0x477a bytes) 000207 1778355288.802538 read 0x028d86 thru 0x036651 (0xd8cc bytes) 000208 1778355288.803452 mapwrite 0x0349da thru 0x03ffff (0xb626 bytes) 000209 1778355288.839171 trunc from 0x040000 to 0x01c713 (0x238ec bytes) 000210 1778355288.848026 trunc from 0x01c714 to 0x013638 (0x90db bytes) 000211 1778355288.856564 read 0x00a66a thru 0x012d9d (0x8734 bytes) 000214 1778355288.859055 mapwrite 0x03fdf9 thru 0x03ffff (0x207 bytes) 000216 1778355288.898907 mapread 0x02bdf1 thru 0x031b6f (0x5d7f bytes) 000217 1778355288.900287 mapread 0x03b4c2 thru 0x03ffff (0x4b3e bytes) 000218 1778355288.901625 read 0x01a25c thru 0x01b9f5 (0x179a bytes) 000222 1778355288.905906 mapread 0x03c0da thru 0x03ffff (0x3f26 bytes) 000224 1778355288.910869 MIRROR_EXTEND to 2 mirrors 000225 1778355289.211556 mapread 0x0081e8 thru 0x00878c (0x5a5 bytes) 000227 1778355289.217001 mapwrite 0x00f494 thru 0x01027e (0xdeb bytes) 000229 1778355289.301988 mapwrite 0x0262a5 thru 0x0359d2 (0xf72e bytes) 000230 1778355289.342037 mapread 0x016fb6 thru 0x02051f (0x956a bytes) 000231 1778355289.348009 read 0x008b78 thru 0x00a24e (0x16d7 bytes) 000232 1778355289.348993 trunc from 0x040000 to 0x00a247 (0x35db8 bytes) 000233 1778355289.360799 trunc from 0x00a248 to 0x0381a5 (0x2df5e bytes) 000234 1778355289.369887 mapread 0x02c223 thru 0x0331ff (0x6fdd bytes) 000235 1778355289.375782 trunc from 0x0381a6 to 0x01b020 (0x1d185 bytes) 000236 1778355289.385665 write 0x004d99 thru 0x004dd1 (0x39 bytes) 000238 1778355289.392791 read 0x0190b5 thru 0x01ac85 (0x1bd1 bytes) 000239 1778355289.398259 mapread 0x00c48e thru 0x011662 (0x51d5 bytes) 000240 1778355289.400660 trunc from 0x01b021 to 0x022e9a (0x7e7a bytes) 000242 1778355289.411738 trunc from 0x022e9b to 0x02cba5 (0x9d0b bytes) 000243 1778355289.423900 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000244 1778355289.496827 mapwrite 0x03f3a9 thru 0x03ffff (0xc57 bytes) 000248 1778355289.527630 trunc from 0x040000 to 0x02d2cf (0x12d30 bytes) 000250 1778355289.534950 trunc from 0x02d2d0 to 0x03c822 (0xf553 bytes) 000254 1778355289.542478 read 0x018b1f thru 0x01b978 (0x2e5a bytes) 000258 1778355289.547587 write 0x0172ed thru 0x024e69 (0xdb7d bytes) 000260 1778355289.553508 mapread 0x0241d3 thru 0x02d4b4 (0x92e2 bytes) 000261 1778355289.558929 trunc from 0x03c823 to 0x033622 (0x9200 bytes) 000262 1778355289.566228 read 0x00b8a1 thru 0x014abd (0x921d bytes) 000263 1778355289.571455 mapwrite 0x020338 thru 0x02e7a2 (0xe46b bytes) 000264 1778355289.596942 mapread 0x02e4bd thru 0x033622 (0x5166 bytes) 000266 1778355289.600768 MIRROR_EXTEND to 2 mirrors 000267 1778355289.869148 mapread 0x00b1e7 thru 0x01596b (0xa785 bytes) 000269 1778355289.882862 MIRROR_EXTEND to 3 mirrors 000270 1778355290.077814 trunc from 0x033623 to 0x01227a (0x213a8 bytes) 000273 1778355290.120979 mapwrite 0x0046f2 thru 0x011065 (0xc974 bytes) 000274 1778355290.163417 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1227b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1227b): Mirror 2: 0x41a86a65 (stale) Mirror 3: 0x41a86a65 (stale) CRC-32 checksum value for chunk [0, 0x1227b): Mirror 1: 0x4f9ef9ac Mirror 4: 0x4f9ef9ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1227b: skipped 000275 1778355290.517754 mapwrite 0x00f38c thru 0x016205 (0x6e7a bytes) 000277 1778355290.634885 read 0x0150c3 thru 0x016205 (0x1143 bytes) 000278 1778355290.635813 mapwrite 0x0366fd thru 0x0393b1 (0x2cb5 bytes) 000280 1778355290.677654 mapwrite 0x02ae0b thru 0x02f9df (0x4bd5 bytes) 000282 1778355290.712247 mapwrite 0x03ea55 thru 0x03ffff (0x15ab bytes) 000283 1778355290.748626 mapwrite 0x0015d3 thru 0x0082e8 (0x6d16 bytes) 000284 1778355290.781086 mapread 0x03e3ae thru 0x03ffff (0x1c52 bytes) 000285 1778355290.783190 mapread 0x038ec8 thru 0x03ffff (0x7138 bytes) 000286 1778355290.785442 mapread 0x014ff3 thru 0x01cea9 (0x7eb7 bytes) 000287 1778355290.793709 MIRROR_RESYNC 4 mirrors 000289 1778355291.128373 read 0x000cd6 thru 0x00690e (0x5c39 bytes) 000292 1778355291.134486 MIRROR_EXTEND to 5 mirrors 000293 1778355291.374226 mapread 0x02d3ca thru 0x03bcb8 (0xe8ef bytes) 000295 1778355291.397357 mapread 0x02dbd3 thru 0x034f57 (0x7385 bytes) 000296 1778355291.404877 write 0x011685 thru 0x012681 (0xffd bytes) 000297 1778355291.449423 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x92915f85 (stale) Mirror 3: 0x92915f85 (stale) Mirror 5: 0x92915f85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x92915f85 (stale) Mirror 3: 0x92915f85 (stale) Mirror 5: 0x92915f85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000298 1778355291.601151 write 0x005144 thru 0x00d4bb (0x8378 bytes) 000299 1778355291.606114 mapread 0x026404 thru 0x02c7b9 (0x63b6 bytes) 000302 1778355291.611939 trunc from 0x040000 to 0x0356e1 (0xa91e bytes) 000303 1778355291.624814 trunc from 0x0356e2 to 0x0382a0 (0x2bbf bytes) 000304 1778355291.631770 trunc from 0x0382a1 to 0x01fb49 (0x18757 bytes) 000305 1778355291.640975 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fb4a): Mirror 2: 0xc2c898bf (stale) Mirror 5: 0xc2c898bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fb4a): Mirror 2: 0xc2c898bf (stale) Mirror 5: 0xc2c898bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fb4a: skipped 000306 1778355291.767777 mapwrite 0x01021d thru 0x01e574 (0xe358 bytes) 000307 1778355291.804279 write 0x0323a1 thru 0x03d4ba (0xb11a bytes) 000308 1778355291.807010 mapwrite 0x01a69f thru 0x028999 (0xe2fb bytes) 000310 1778355291.836142 mapread 0x0249f4 thru 0x02fb41 (0xb14e bytes) 000311 1778355291.845558 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d4bb): Mirror 5: 0x1659fced (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4bb): Mirror 5: 0x1659fced (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4bb: skipped 000312 1778355291.976982 mapwrite 0x00a8d9 thru 0x018f47 (0xe66f bytes) 000314 1778355292.020749 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d4bb): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4bb): Mirror 5: 0x1659fced (stale) CRC-32 checksum value for chunk [0, 0x3d4bb): Mirror 1: 0x88236ef0 Mirror 6: 0x88236ef0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4bb: skipped 000316 1778355292.403342 mapread 0x030daa thru 0x031ee6 (0x113d bytes) 000318 1778355292.410120 read 0x023b34 thru 0x0311c0 (0xd68d bytes) 000319 1778355292.414721 mapread 0x0133cb thru 0x022d09 (0xf93f bytes) 000323 1778355292.426464 mapwrite 0x01e3f9 thru 0x025a5b (0x7663 bytes) 000324 1778355292.503283 write 0x01e9bf thru 0x02d7f1 (0xee33 bytes) 000325 1778355292.507174 mapread 0x0294de thru 0x031e1d (0x8940 bytes) 000327 1778355292.512624 mapread 0x016039 thru 0x017e14 (0x1ddc bytes) 000328 1778355292.516882 mapwrite 0x01935c thru 0x01e581 (0x5226 bytes) 000330 1778355292.537275 trunc from 0x03d4bb to 0x00afbd (0x324fd bytes) 000331 1778355292.545315 read 0x002457 thru 0x002885 (0x42f bytes) 000332 1778355292.548907 mapread 0x001e89 thru 0x0050d6 (0x324e bytes) 000334 1778355292.550160 write 0x03eb88 thru 0x03ffff (0x1478 bytes) 000336 1778355292.551691 mapread 0x01373a thru 0x01383c (0x103 bytes) 000338 1778355292.556683 trunc from 0x040000 to 0x0308ec (0xf713 bytes) 000339 1778355292.563269 mapread 0x016acc thru 0x01b2b8 (0x47ed bytes) 000341 1778355292.564857 mapread 0x02006a thru 0x02b87e (0xb815 bytes) 000343 1778355292.568673 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x308ed): Mirror 6: 0xced4fd6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x308ed): Mirror 6: 0xced4fd6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x308ed: skipped 000344 1778355292.722364 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x308ed): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x308ed): Mirror 6: 0xced4fd6a (stale) CRC-32 checksum value for chunk [0, 0x308ed): Mirror 1: 0xab7965fc Mirror 7: 0xab7965fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x308ed: skipped 000345 1778355293.064194 write 0x0343d8 thru 0x03d723 (0x934c bytes) 000346 1778355293.094535 read 0x002edf thru 0x00e2b0 (0xb3d2 bytes) 000349 1778355293.104904 MIRROR_RESYNC 3 mirrors 000350 1778355293.443419 read 0x00dc88 thru 0x00eb03 (0xe7c bytes) 000351 1778355293.447032 mapread 0x0177b3 thru 0x025904 (0xe152 bytes) 000353 1778355293.464109 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000355 1778355293.616195 MIRROR_EXTEND to 3 mirrors 000357 1778355293.928696 read 0x0263c1 thru 0x02e88d (0x84cd bytes) 000358 1778355293.933902 mapread 0x014a6f thru 0x014dee (0x380 bytes) 000359 1778355293.937025 read 0x02c162 thru 0x03bf54 (0xfdf3 bytes) 000362 1778355293.943424 write 0x01358c thru 0x022523 (0xef98 bytes) 000363 1778355293.987959 mapread 0x020b09 thru 0x02527d (0x4775 bytes) 000364 1778355293.996625 mapwrite 0x00cd3d thru 0x01123c (0x4500 bytes) 000365 1778355294.039708 mapwrite 0x039ae4 thru 0x03fd1e (0x623b bytes) 000366 1778355294.098662 read 0x02e49e thru 0x031364 (0x2ec7 bytes) 000369 1778355294.105476 trunc from 0x03fd1f to 0x02553a (0x1a7e4 bytes) 000370 1778355294.116364 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2553b): Mirror 7: 0xab1f22ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2553b): Mirror 7: 0xab1f22ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2553b: skipped 000371 1778355294.245206 mapread 0x0200b3 thru 0x02553a (0x5488 bytes) 000372 1778355294.257399 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2553b): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2553b): Mirror 7: 0xab1f22ab (stale) CRC-32 checksum value for chunk [0, 0x2553b): Mirror 1: 0xbef1ba5f Mirror 8: 0xbef1ba5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2553b: skipped 000375 1778355294.576967 read 0x015d2a thru 0x020d9b (0xb072 bytes) 000376 1778355294.582359 trunc from 0x02553b to 0x003e9e (0x2169c bytes) 000377 1778355294.631731 trunc from 0x003e9f to 0x03265e (0x2e7c0 bytes) 000380 1778355294.644377 write 0x001d1b thru 0x006ba2 (0x4e88 bytes) 000381 1778355294.653874 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3265f): Mirror 8: 0xc5917b19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3265f): Mirror 8: 0xc5917b19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3265f: skipped 000384 1778355294.811316 write 0x02e40b thru 0x039f45 (0xbb3b bytes) 000386 1778355294.818323 mapwrite 0x03bfa9 thru 0x03e5fd (0x2655 bytes) 000387 1778355294.853345 MIRROR_RESYNC 2 mirrors 000388 1778355295.125151 mapread 0x017092 thru 0x018885 (0x17f4 bytes) 000389 1778355295.131845 mapread 0x0073e7 thru 0x0074fc (0x116 bytes) 000391 1778355295.140290 MIRROR_RESYNC 2 mirrors 000393 1778355295.317042 mapwrite 0x0066b7 thru 0x0070df (0xa29 bytes) 000395 1778355295.389260 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e5fe): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e5fe): Mirror 8: 0x8684f2d6 (stale) CRC-32 checksum value for chunk [0, 0x3e5fe): Mirror 1: 0xba8eb4b1 Mirror 9: 0xba8eb4b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e5fe: skipped 000396 1778355295.811758 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000397 1778355295.956824 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000398 1778355296.010104 write 0x02bf63 thru 0x02dbfe (0x1c9c bytes) 000399 1778355296.017675 write 0x01f9e1 thru 0x0278e8 (0x7f08 bytes) 000400 1778355296.022280 read 0x002863 thru 0x0031e0 (0x97e bytes) 000403 1778355296.028307 MIRROR_EXTEND to 2 mirrors 000406 1778355296.275976 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000408 1778355296.318157 mapwrite 0x031119 thru 0x035ed4 (0x4dbc bytes) 000410 1778355296.341343 mapwrite 0x013ccd thru 0x01ccec (0x9020 bytes) 000411 1778355296.363850 mapread 0x0371ef thru 0x037c4b (0xa5d bytes) 000412 1778355296.364781 trunc from 0x03e5fe to 0x02a302 (0x142fb bytes) 000414 1778355296.378386 read 0x010aa6 thru 0x019f09 (0x9464 bytes) 000416 1778355296.380281 trunc from 0x02a303 to 0x013490 (0x16e72 bytes) 000418 1778355296.388158 write 0x0338ce thru 0x03690b (0x303e bytes) 000421 1778355296.390736 trunc from 0x03690c to 0x017188 (0x1f783 bytes) 000422 1778355296.398507 mapread 0x014322 thru 0x017188 (0x2e67 bytes) 000423 1778355296.403273 mapread 0x00c5d4 thru 0x013dd4 (0x7801 bytes) 000424 1778355296.404732 trunc from 0x017189 to 0x028d41 (0x11bb9 bytes) 000426 1778355296.415176 read 0x022aec thru 0x028d41 (0x6256 bytes) 000428 1778355296.422114 read 0x01f20a thru 0x02203a (0x2e31 bytes) 000429 1778355296.422995 mapread 0x01f019 thru 0x0206af (0x1697 bytes) 000431 1778355296.425025 trunc from 0x028d42 to 0x002f70 (0x25dd1 bytes) 000434 1778355296.437306 write 0x003e79 thru 0x005e84 (0x200c bytes) 000440 1778355296.443072 read 0x000aa2 thru 0x005e84 (0x53e3 bytes) 000443 1778355296.445817 read 0x005b40 thru 0x005e84 (0x345 bytes) 000444 1778355296.446610 trunc from 0x005e85 to 0x00ac87 (0x4e03 bytes) 000446 1778355296.456148 write 0x0235ca thru 0x02755a (0x3f91 bytes) 000448 1778355296.458542 mapwrite 0x023802 thru 0x02840c (0x4c0b bytes) 000449 1778355296.503444 read 0x008fcc thru 0x017818 (0xe84d bytes) 000452 1778355296.506079 mapwrite 0x00a419 thru 0x00dcde (0x38c6 bytes) 000455 1778355296.532015 trunc from 0x02840d to 0x01c972 (0xba9a bytes) 000458 1778355296.544131 trunc from 0x01c973 to 0x0194a6 (0x34cc bytes) 000459 1778355296.553380 mapwrite 0x013757 thru 0x021fec (0xe896 bytes) 000463 1778355296.594861 mapwrite 0x014ad7 thru 0x01a735 (0x5c5f bytes) 000464 1778355296.615512 write 0x036542 thru 0x03ffff (0x9abe bytes) 000466 1778355296.617279 mapread 0x0020c0 thru 0x008a69 (0x69aa bytes) 000467 1778355296.618292 write 0x033a31 thru 0x035c1c (0x21ec bytes) 000470 1778355296.622814 mapread 0x03812e thru 0x03f26f (0x7142 bytes) 000471 1778355296.623682 read 0x03864a thru 0x03c058 (0x3a0f bytes) 000474 1778355296.625017 mapread 0x01bbd5 thru 0x02094a (0x4d76 bytes) 000477 1778355296.627239 mapread 0x01a50c thru 0x01d669 (0x315e bytes) 000479 1778355296.628822 mapwrite 0x019d78 thru 0x01b94b (0x1bd4 bytes) 000483 1778355296.656022 mapread 0x025c41 thru 0x0277c8 (0x1b88 bytes) 000485 1778355296.657974 read 0x00cbfa thru 0x016fa2 (0xa3a9 bytes) 000487 1778355296.659448 read 0x03b902 thru 0x03fea5 (0x45a4 bytes) 000489 1778355296.660970 write 0x02ff62 thru 0x032ea9 (0x2f48 bytes) 000490 1778355296.662216 mapread 0x022bec thru 0x032509 (0xf91e bytes) 000491 1778355296.663787 write 0x03754e thru 0x03ffff (0x8ab2 bytes) 000493 1778355296.665761 trunc from 0x040000 to 0x003f27 (0x3c0d8 bytes) 000495 1778355296.677204 read 0x002042 thru 0x003f27 (0x1ee6 bytes) 000497 1778355296.679081 mapread 0x000f26 thru 0x003f27 (0x3002 bytes) 000498 1778355296.680390 mapread 0x0025f5 thru 0x003f27 (0x1933 bytes) 000501 1778355296.683894 read 0x0039b2 thru 0x003f27 (0x576 bytes) 000502 1778355296.684898 write 0x0140e3 thru 0x015cfb (0x1c19 bytes) 000503 1778355296.686212 write 0x022593 thru 0x02663e (0x40ac bytes) 000504 1778355296.690827 MIRROR_EXTEND to 2 mirrors 000507 1778355296.989511 write 0x03ec6c thru 0x03ffff (0x1394 bytes) 000509 1778355297.019481 read 0x00d680 thru 0x01acb3 (0xd634 bytes) 000510 1778355297.024959 read 0x0047c2 thru 0x00e490 (0x9ccf bytes) 000511 1778355297.030251 mapread 0x0213f6 thru 0x0311ee (0xfdf9 bytes) 000512 1778355297.031962 read 0x020250 thru 0x027608 (0x73b9 bytes) 000513 1778355297.032788 mapread 0x01bd6c thru 0x01ca0d (0xca2 bytes) 000514 1778355297.033972 trunc from 0x040000 to 0x022646 (0x1d9b9 bytes) 000515 1778355297.052449 trunc from 0x022647 to 0x00f260 (0x133e6 bytes) 000518 1778355297.062088 read 0x001352 thru 0x003ab3 (0x2762 bytes) 000520 1778355297.063374 trunc from 0x00f261 to 0x007587 (0x7cd9 bytes) 000521 1778355297.071329 trunc from 0x007588 to 0x024170 (0x1cbe9 bytes) 000523 1778355297.079599 write 0x02da04 thru 0x03202e (0x462b bytes) 000527 1778355297.085183 MIRROR_RESYNC 2 mirrors 000528 1778355297.385847 mapwrite 0x02b579 thru 0x02fdae (0x4836 bytes) 000530 1778355297.466324 read 0x017142 thru 0x01c929 (0x57e8 bytes) 000531 1778355297.471354 trunc from 0x03202f to 0x000967 (0x316c7 bytes) 000533 1778355297.493103 write 0x03ec8f thru 0x03ffff (0x1371 bytes) 000536 1778355297.496158 trunc from 0x040000 to 0x02a66b (0x15994 bytes) 000537 1778355297.505851 mapwrite 0x0128ab thru 0x01c4d2 (0x9c28 bytes) 000539 1778355297.544359 mapwrite 0x02a2b5 thru 0x02ec8d (0x49d9 bytes) 000540 1778355297.578503 mapread 0x02b0fa thru 0x02ec8d (0x3b94 bytes) 000541 1778355297.580789 write 0x0341f5 thru 0x03bb6e (0x797a bytes) 000542 1778355297.582498 write 0x029054 thru 0x02be56 (0x2e03 bytes) 000546 1778355297.587420 read 0x0100ba thru 0x01168e (0x15d5 bytes) 000549 1778355297.589679 write 0x0328f4 thru 0x03ffff (0xd70c bytes) 000550 1778355297.596561 trunc from 0x040000 to 0x008d6c (0x37293 bytes) 000552 1778355297.605680 mapread 0x002cc3 thru 0x008d6c (0x60aa bytes) 000553 1778355297.606846 write 0x0221be thru 0x02c601 (0xa444 bytes) 000554 1778355297.608312 read 0x02571f thru 0x0259bc (0x29e bytes) 000556 1778355297.611900 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000558 1778355297.685409 mapread 0x017cb9 thru 0x01ea91 (0x6dd9 bytes) 000560 1778355297.700391 MIRROR_EXTEND to 2 mirrors 000562 1778355297.968508 trunc from 0x02c602 to 0x00a49f (0x22162 bytes) 000565 1778355298.020133 mapwrite 0x03c1ce thru 0x03e413 (0x2246 bytes) 000566 1778355298.065809 mapread 0x017b18 thru 0x018419 (0x902 bytes) 000567 1778355298.074905 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e414): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e414): Mirror 2: 0xbd3d2a01 (stale) CRC-32 checksum value for chunk [0, 0x3e414): Mirror 1: 0x1e16d642 Mirror 3: 0x1e16d642 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e414: skipped 000568 1778355298.415295 read 0x032f0e thru 0x03a81a (0x790d bytes) 000570 1778355298.420800 write 0x005347 thru 0x0144b4 (0xf16e bytes) 000571 1778355298.463403 mapread 0x015ae1 thru 0x0203ec (0xa90c bytes) 000572 1778355298.474714 mapread 0x02d405 thru 0x02faa9 (0x26a5 bytes) 000575 1778355298.482317 write 0x0039ef thru 0x013090 (0xf6a2 bytes) 000576 1778355298.489216 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e414): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e414): Mirror 2: 0xbd3d2a01 (stale) Mirror 3: 0x1e16d642 (stale) CRC-32 checksum value for chunk [0, 0x3e414): Mirror 1: 0x17d86ffd Mirror 4: 0x17d86ffd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e414: skipped 000578 1778355298.806894 write 0x024359 thru 0x0290a3 (0x4d4b bytes) 000579 1778355298.848095 read 0x035f35 thru 0x036474 (0x540 bytes) 000580 1778355298.851763 mapwrite 0x00f6a6 thru 0x01225c (0x2bb7 bytes) 000581 1778355298.884083 mapread 0x01bb18 thru 0x0211df (0x56c8 bytes) 000582 1778355298.890776 read 0x02ea49 thru 0x03c92b (0xdee3 bytes) 000584 1778355298.897086 mapwrite 0x0060bb thru 0x0102d4 (0xa21a bytes) 000588 1778355298.928044 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e414): Mirror 2: 0xbd3d2a01 (stale) Mirror 3: 0x1e16d642 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e414): Mirror 2: 0xbd3d2a01 (stale) Mirror 3: 0x1e16d642 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e414: skipped 000589 1778355299.086315 read 0x0393f8 thru 0x03e413 (0x501c bytes) 000590 1778355299.091709 write 0x011ac5 thru 0x01d866 (0xbda2 bytes) 000592 1778355299.099015 mapwrite 0x01c622 thru 0x02714f (0xab2e bytes) 000593 1778355299.136103 write 0x00931f thru 0x0099aa (0x68c bytes) 000594 1778355299.142038 read 0x013314 thru 0x01a65c (0x7349 bytes) 000599 1778355299.148371 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e414): Mirror 3: 0x1e16d642 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e414): Mirror 3: 0x1e16d642 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e414: skipped 000600 1778355299.289676 MIRROR_RESYNC 2 mirrors 000601 1778355299.590740 mapread 0x01d212 thru 0x02c0cd (0xeebc bytes) 000602 1778355299.606017 trunc from 0x03e414 to 0x0298f1 (0x14b22 bytes) 000604 1778355299.646418 trunc from 0x0298f2 to 0x00de25 (0x1bacc bytes) 000606 1778355299.655597 mapread 0x00051d thru 0x001cc9 (0x17ad bytes) 000611 1778355299.665727 MIRROR_RESYNC 2 mirrors 000612 1778355299.889808 read 0x0008af thru 0x005ee1 (0x5633 bytes) 000613 1778355299.893477 MIRROR_RESYNC 2 mirrors 000615 1778355300.029877 mapwrite 0x022de0 thru 0x02e0cd (0xb2ee bytes) 000616 1778355300.108107 mapread 0x00010c thru 0x00948a (0x937f bytes) 000619 1778355300.114003 mapread 0x00cec5 thru 0x00e471 (0x15ad bytes) 000620 1778355300.117171 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e0ce): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0ce): Mirror 3: 0xd357602e (stale) CRC-32 checksum value for chunk [0, 0x2e0ce): Mirror 1: 0xce4a18f3 Mirror 4: 0xce4a18f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0ce: skipped 000621 1778355300.460754 MIRROR_RESYNC 3 mirrors 000623 1778355300.740535 write 0x006a7c thru 0x01585c (0xede1 bytes) 000624 1778355300.779956 read 0x000bc6 thru 0x0082b4 (0x76ef bytes) 000625 1778355300.785431 mapread 0x02970f thru 0x02e0cd (0x49bf bytes) 000626 1778355300.794925 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e0ce): Mirror 4: 0xce4a18f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0ce): Mirror 4: 0xce4a18f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0ce: skipped 000627 1778355300.934799 mapwrite 0x026330 thru 0x033aef (0xd7c0 bytes) 000630 1778355300.995827 MIRROR_RESYNC 2 mirrors 000631 1778355301.278971 read 0x031edf thru 0x033aef (0x1c11 bytes) 000632 1778355301.282703 mapwrite 0x01aafb thru 0x01cfc4 (0x24ca bytes) 000633 1778355301.359961 read 0x00ce1c thru 0x012524 (0x5709 bytes) 000634 1778355301.366862 trunc from 0x033af0 to 0x037371 (0x3882 bytes) 000636 1778355301.379795 read 0x015968 thru 0x022f94 (0xd62d bytes) 000638 1778355301.381673 mapread 0x025382 thru 0x033be1 (0xe860 bytes) 000639 1778355301.383514 mapwrite 0x025d0b thru 0x026fb6 (0x12ac bytes) 000640 1778355301.406997 read 0x036a37 thru 0x037371 (0x93b bytes) 000641 1778355301.411488 mapwrite 0x014aff thru 0x01e3eb (0x98ed bytes) 000642 1778355301.438384 mapwrite 0x031e51 thru 0x03a38b (0x853b bytes) 000643 1778355301.473931 write 0x007ed6 thru 0x017930 (0xfa5b bytes) 000644 1778355301.475565 read 0x00253c thru 0x00bfc9 (0x9a8e bytes) 000646 1778355301.479308 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a38c): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a38c): Mirror 4: 0x4f2b62a3 (stale) CRC-32 checksum value for chunk [0, 0x3a38c): Mirror 1: 0x2c5211f3 Mirror 5: 0x2c5211f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a38c: skipped 000647 1778355301.852280 mapwrite 0x023aa1 thru 0x0247ba (0xd1a bytes) 000648 1778355301.935932 read 0x036a9c thru 0x037080 (0x5e5 bytes) 000652 1778355301.942997 read 0x015f9a thru 0x01ec4d (0x8cb4 bytes) 000653 1778355301.949761 write 0x01ad73 thru 0x01fb8d (0x4e1b bytes) 000654 1778355301.951310 mapwrite 0x030926 thru 0x03ffff (0xf6da bytes) 000655 1778355302.004466 mapread 0x031fbc thru 0x038eb8 (0x6efd bytes) 000656 1778355302.005520 mapwrite 0x0201d6 thru 0x02e38e (0xe1b9 bytes) 000657 1778355302.026194 mapwrite 0x01213f thru 0x0190c3 (0x6f85 bytes) 000658 1778355302.048651 trunc from 0x040000 to 0x02c32e (0x13cd1 bytes) 000659 1778355302.058564 write 0x0353b6 thru 0x03fa05 (0xa650 bytes) 000660 1778355302.060850 trunc from 0x03fa06 to 0x029ffe (0x15a07 bytes) 000663 1778355302.073281 mapread 0x019fe4 thru 0x021e24 (0x7e41 bytes) 000664 1778355302.077823 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29fff): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29fff): Mirror 4: 0xeb82ff90 (stale) Mirror 5: 0xf8cbee34 (stale) CRC-32 checksum value for chunk [0, 0x29fff): Mirror 1: 0xfb280057 Mirror 6: 0xfb280057 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29fff: skipped 000667 1778355302.485718 read 0x027293 thru 0x029ffe (0x2d6c bytes) 000668 1778355302.490886 write 0x03e2ec thru 0x03ffff (0x1d14 bytes) 000669 1778355302.525839 read 0x01a777 thru 0x0279db (0xd265 bytes) 000673 1778355302.534575 write 0x013cfc thru 0x017aa9 (0x3dae bytes) 000674 1778355302.535930 read 0x028d3e thru 0x02ffdb (0x729e bytes) 000675 1778355302.536745 mapwrite 0x039d0d thru 0x03f6e3 (0x59d7 bytes) 000676 1778355302.564379 mapwrite 0x00c4bb thru 0x01775d (0xb2a3 bytes) 000679 1778355302.591425 trunc from 0x040000 to 0x039eb9 (0x6146 bytes) 000682 1778355302.616706 MIRROR_RESYNC 4 mirrors 000683 1778355302.999755 write 0x03e17b thru 0x03ffff (0x1e85 bytes) 000686 1778355303.033339 mapwrite 0x023b51 thru 0x032cdc (0xf18c bytes) 000687 1778355303.079560 trunc from 0x040000 to 0x03c5a6 (0x3a59 bytes) 000688 1778355303.101778 write 0x01683f thru 0x01cfd9 (0x679b bytes) 000690 1778355303.108407 mapwrite 0x018812 thru 0x01cd88 (0x4577 bytes) 000691 1778355303.133484 read 0x01c3eb thru 0x02afc2 (0xebd8 bytes) 000692 1778355303.134409 mapread 0x00bd4e thru 0x00caae (0xd61 bytes) 000693 1778355303.140355 read 0x01ff83 thru 0x025e85 (0x5f03 bytes) 000694 1778355303.141194 mapwrite 0x00ea98 thru 0x0197cc (0xad35 bytes) 000696 1778355303.169324 trunc from 0x03c5a7 to 0x00b0a5 (0x31501 bytes) 000697 1778355303.179391 write 0x001985 thru 0x006475 (0x4af1 bytes) 000698 1778355303.183893 write 0x0372c6 thru 0x03ffff (0x8d3a bytes) 000699 1778355303.185735 trunc from 0x040000 to 0x01af66 (0x25099 bytes) 000700 1778355303.198455 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1af67): Mirror 5: 0x891e858c (stale) Mirror 6: 0x891e858c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1af67): Mirror 5: 0x891e858c (stale) Mirror 6: 0x891e858c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1af67: skipped 000701 1778355303.356115 read 0x01911f thru 0x01af66 (0x1e48 bytes) 000704 1778355303.362207 mapwrite 0x0108b4 thru 0x01a195 (0x98e2 bytes) 000705 1778355303.403730 write 0x01a5a0 thru 0x0209ed (0x644e bytes) 000708 1778355303.406676 write 0x03d195 thru 0x03de1e (0xc8a bytes) 000709 1778355303.407901 mapwrite 0x015a65 thru 0x020e8f (0xb42b bytes) 000710 1778355303.437952 mapwrite 0x01dfcb thru 0x023953 (0x5989 bytes) 000712 1778355303.467153 trunc from 0x03de1f to 0x02f0de (0xed40 bytes) 000713 1778355303.477490 mapread 0x00c995 thru 0x01ae55 (0xe4c1 bytes) 000716 1778355303.480558 write 0x000889 thru 0x00c12d (0xb8a5 bytes) 000717 1778355303.487178 write 0x02d7c8 thru 0x02fbcb (0x2404 bytes) 000719 1778355303.493890 mapwrite 0x00515d thru 0x00ff23 (0xadc7 bytes) 000720 1778355303.522814 write 0x02fed1 thru 0x03e63a (0xe76a bytes) 000722 1778355303.525482 write 0x03674d thru 0x03ffff (0x98b3 bytes) 000723 1778355303.530027 MIRROR_RESYNC 3 mirrors 000725 1778355303.888946 MIRROR_RESYNC 3 mirrors 000729 1778355304.073726 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000730 1778355304.228200 mapread 0x003a6e thru 0x009176 (0x5709 bytes) 000732 1778355304.241022 read 0x002573 thru 0x00c3a0 (0x9e2e bytes) 000733 1778355304.242361 read 0x03291e thru 0x03e82a (0xbf0d bytes) 000735 1778355304.252673 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000736 1778355304.325439 write 0x00fead thru 0x00ff61 (0xb5 bytes) 000737 1778355304.334854 trunc from 0x040000 to 0x020ffb (0x1f004 bytes) 000741 1778355304.355494 read 0x01656f thru 0x020ffb (0xaa8d bytes) 000742 1778355304.361817 read 0x004680 thru 0x00c1f7 (0x7b78 bytes) 000743 1778355304.368190 read 0x003194 thru 0x00f670 (0xc4dd bytes) 000744 1778355304.369166 mapwrite 0x02e4ca thru 0x02eaff (0x636 bytes) 000745 1778355304.409635 mapread 0x00edc3 thru 0x01965b (0xa899 bytes) 000747 1778355304.411865 mapwrite 0x0016fd thru 0x006b98 (0x549c bytes) 000748 1778355304.435954 write 0x0037de thru 0x00819e (0x49c1 bytes) 000753 1778355304.440152 mapread 0x01bb20 thru 0x022173 (0x6654 bytes) 000754 1778355304.441407 trunc from 0x02eb00 to 0x037edf (0x93e0 bytes) 000755 1778355304.450961 mapread 0x0366db thru 0x037edf (0x1805 bytes) 000756 1778355304.458924 MIRROR_EXTEND to 2 mirrors 000757 1778355304.750206 mapwrite 0x00d68c thru 0x018e1e (0xb793 bytes) 000759 1778355304.844484 mapwrite 0x01623c thru 0x0164ea (0x2af bytes) 000760 1778355304.866895 trunc from 0x037ee0 to 0x024441 (0x13a9e bytes) 000761 1778355304.888831 write 0x013dc8 thru 0x016d21 (0x2f5a bytes) 000763 1778355304.894861 MIRROR_RESYNC 2 mirrors 000764 1778355305.216181 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000767 1778355305.287897 read 0x022133 thru 0x024441 (0x230f bytes) 000768 1778355305.292859 mapread 0x023d60 thru 0x024441 (0x6e2 bytes) 000769 1778355305.295025 trunc from 0x024442 to 0x0250d1 (0xc90 bytes) 000775 1778355305.312378 trunc from 0x0250d2 to 0x037489 (0x123b8 bytes) 000777 1778355305.322906 trunc from 0x03748a to 0x00a74f (0x2cd3a bytes) 000778 1778355305.331897 read 0x002b82 thru 0x006d26 (0x41a5 bytes) 000779 1778355305.337669 trunc from 0x00a750 to 0x01e008 (0x138b9 bytes) 000780 1778355305.346432 mapwrite 0x0085c5 thru 0x017259 (0xec95 bytes) 000782 1778355305.379204 mapread 0x006b65 thru 0x00a3e3 (0x387f bytes) 000783 1778355305.380241 mapwrite 0x00cf97 thru 0x014df0 (0x7e5a bytes) 000785 1778355305.404946 write 0x006a12 thru 0x014005 (0xd5f4 bytes) 000786 1778355305.408027 write 0x02e2d6 thru 0x02e7c7 (0x4f2 bytes) 000787 1778355305.410201 trunc from 0x02e7c8 to 0x02dc67 (0xb60 bytes) 000788 1778355305.421137 trunc from 0x02dc68 to 0x03125c (0x35f5 bytes) 000790 1778355305.431251 trunc from 0x03125d to 0x00c164 (0x250f8 bytes) 000793 1778355305.444481 MIRROR_EXTEND to 2 mirrors 000794 1778355305.743025 write 0x02e1f2 thru 0x03e144 (0xff53 bytes) 000795 1778355305.780377 mapwrite 0x011323 thru 0x014663 (0x3341 bytes) 000796 1778355305.819080 write 0x019340 thru 0x01d81a (0x44db bytes) 000797 1778355305.823413 trunc from 0x03e145 to 0x011d16 (0x2c42e bytes) 000798 1778355305.845176 write 0x032bd1 thru 0x037f2e (0x535e bytes) 000799 1778355305.846817 read 0x02a874 thru 0x02d34e (0x2adb bytes) 000801 1778355305.851764 read 0x0214b5 thru 0x021686 (0x1d2 bytes) 000806 1778355305.860699 mapread 0x00697b thru 0x0167ca (0xfe50 bytes) 000807 1778355305.862437 trunc from 0x037f2f to 0x035def (0x213f bytes) 000808 1778355305.871740 read 0x00f3ae thru 0x012f1f (0x3b72 bytes) 000809 1778355305.872801 read 0x012e01 thru 0x01438a (0x158a bytes) 000810 1778355305.873744 read 0x01b00f thru 0x01edf9 (0x3deb bytes) 000814 1778355305.877022 read 0x01d855 thru 0x02c44a (0xebf6 bytes) 000816 1778355305.881714 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000818 1778355305.963080 mapread 0x031691 thru 0x035def (0x475f bytes) 000819 1778355305.975288 mapread 0x00935f thru 0x018bb5 (0xf857 bytes) 000820 1778355305.986419 write 0x02d6b0 thru 0x03183c (0x418d bytes) 000821 1778355305.987922 write 0x013ecd thru 0x020617 (0xc74b bytes) 000823 1778355305.993419 mapread 0x02588f thru 0x0303ae (0xab20 bytes) 000824 1778355305.999825 mapwrite 0x000071 thru 0x005d20 (0x5cb0 bytes) 000826 1778355306.030790 trunc from 0x035df0 to 0x002cd5 (0x3311a bytes) 000827 1778355306.041106 mapwrite 0x00f7b5 thru 0x011a89 (0x22d5 bytes) 000831 1778355306.079792 trunc from 0x011a8a to 0x02f3b0 (0x1d927 bytes) 000832 1778355306.087859 mapwrite 0x02fc1b thru 0x038a62 (0x8e48 bytes) 000833 1778355306.129008 read 0x017380 thru 0x024a61 (0xd6e2 bytes) 000834 1778355306.134416 mapwrite 0x035cb8 thru 0x03f5c0 (0x9909 bytes) 000835 1778355306.174307 mapwrite 0x03f449 thru 0x03ffff (0xbb7 bytes) 000843 1778355306.208314 write 0x030b7a thru 0x030b9c (0x23 bytes) 000845 1778355306.210285 write 0x0316e0 thru 0x03a313 (0x8c34 bytes) 000846 1778355306.211738 read 0x02d35a thru 0x034d68 (0x7a0f bytes) 000848 1778355306.213187 read 0x02deb5 thru 0x03d2ee (0xf43a bytes) 000849 1778355306.214006 read 0x021d4b thru 0x02bc19 (0x9ecf bytes) 000852 1778355306.221714 mapwrite 0x00f28d thru 0x01c3b5 (0xd129 bytes) 000853 1778355306.249986 write 0x00d0be thru 0x014339 (0x727c bytes) 000856 1778355306.253241 trunc from 0x040000 to 0x02c28e (0x13d71 bytes) 000857 1778355306.262903 mapwrite 0x004ac0 thru 0x0128eb (0xde2c bytes) 000858 1778355306.291997 trunc from 0x02c28f to 0x02aa94 (0x17fa bytes) 000859 1778355306.301284 write 0x027144 thru 0x02ea57 (0x7914 bytes) 000863 1778355306.305571 mapwrite 0x00425a thru 0x00fabf (0xb866 bytes) 000868 1778355306.335636 read 0x00075c thru 0x00be91 (0xb736 bytes) 000869 1778355306.336801 write 0x00968f thru 0x013798 (0xa10a bytes) 000870 1778355306.338632 mapwrite 0x00fadc thru 0x01ab3a (0xb05f bytes) 000872 1778355306.367434 mapread 0x029ea4 thru 0x02ea57 (0x4bb4 bytes) 000875 1778355306.371186 mapread 0x018d0e thru 0x01add5 (0x20c8 bytes) 000882 1778355306.378074 mapwrite 0x013a29 thru 0x0175db (0x3bb3 bytes) 000883 1778355306.397516 mapread 0x025ae4 thru 0x02ea57 (0x8f74 bytes) 000885 1778355306.400902 mapwrite 0x00f9af thru 0x015679 (0x5ccb bytes) 000886 1778355306.422866 mapwrite 0x013c90 thru 0x016b0e (0x2e7f bytes) 000887 1778355306.445967 mapread 0x00c6d1 thru 0x01b77f (0xf0af bytes) 000888 1778355306.447648 read 0x01d878 thru 0x022c5b (0x53e4 bytes) 000891 1778355306.449796 mapwrite 0x03ad6d thru 0x03d8cf (0x2b63 bytes) 000894 1778355306.488860 mapread 0x026ac7 thru 0x02dadc (0x7016 bytes) 000896 1778355306.491104 read 0x009843 thru 0x01162c (0x7dea bytes) 000897 1778355306.491982 mapwrite 0x017f75 thru 0x025b91 (0xdc1d bytes) 000899 1778355306.523485 MIRROR_EXTEND to 2 mirrors 000900 1778355306.818525 write 0x03ed52 thru 0x03ffff (0x12ae bytes) 000901 1778355306.855296 mapread 0x036b44 thru 0x03ffff (0x94bc bytes) 000902 1778355306.867329 read 0x015f17 thru 0x01e5d0 (0x86ba bytes) 000903 1778355306.875426 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x302a17ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xed312d60 Mirror 3: 0xed312d60 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000905 1778355307.289114 trunc from 0x040000 to 0x00f0be (0x30f41 bytes) 000909 1778355307.336185 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xf0bf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf0bf): Mirror 2: 0x10fde7f8 (stale) Mirror 3: 0x10fde7f8 (stale) CRC-32 checksum value for chunk [0, 0xf0bf): Mirror 1: 0x10fde7f8 Mirror 4: 0x10fde7f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf0bf: skipped 000910 1778355307.711744 write 0x032404 thru 0x03ffe1 (0xdbde bytes) 000911 1778355307.749823 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ffe2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffe2): Mirror 2: 0x9e6a6453 (stale) Mirror 3: 0xb4e6ce1a (stale) Mirror 4: 0xa77f62c1 (stale) CRC-32 checksum value for chunk [0, 0x3ffe2): Mirror 1: 0xaa26f80f Mirror 5: 0xaa26f80f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffe2: skipped 000912 1778355308.203869 read 0x00a58b thru 0x00ef62 (0x49d8 bytes) 000913 1778355308.215958 MIRROR_RESYNC 5 mirrors 000914 1778355308.615334 trunc from 0x03ffe2 to 0x022d21 (0x1d2c0 bytes) 000915 1778355308.648986 write 0x027cb7 thru 0x03019e (0x84e8 bytes) 000917 1778355308.650567 read 0x01a356 thru 0x01af2f (0xbda bytes) 000918 1778355308.653031 read 0x02b32e thru 0x030160 (0x4e33 bytes) 000921 1778355308.654418 write 0x01cd8e thru 0x02be00 (0xf073 bytes) 000922 1778355308.660381 MIRROR_RESYNC 5 mirrors 000923 1778355309.005552 mapwrite 0x017991 thru 0x02333f (0xb9af bytes) 000925 1778355309.093820 write 0x030b75 thru 0x0386d2 (0x7b5e bytes) 000926 1778355309.098673 write 0x0229ff thru 0x023522 (0xb24 bytes) 000927 1778355309.099692 trunc from 0x0386d3 to 0x027ee6 (0x107ec bytes) 000929 1778355309.119637 mapread 0x01f592 thru 0x027ee6 (0x8955 bytes) 000930 1778355309.120961 write 0x01e7f6 thru 0x02761c (0x8e27 bytes) 000931 1778355309.122156 mapwrite 0x00d0d2 thru 0x00f688 (0x25b7 bytes) 000932 1778355309.150344 trunc from 0x027ee7 to 0x01efe8 (0x8efe bytes) 000933 1778355309.159373 mapwrite 0x015cf0 thru 0x01b426 (0x5737 bytes) 000935 1778355309.184287 mapread 0x00df57 thru 0x01361e (0x56c8 bytes) 000936 1778355309.187935 MIRROR_RESYNC 5 mirrors 000937 1778355309.501425 mapread 0x0194aa thru 0x01efe8 (0x5b3f bytes) 000939 1778355309.507860 read 0x0130bc thru 0x01d30d (0xa252 bytes) 000940 1778355309.509998 write 0x028de2 thru 0x037b9c (0xedbb bytes) 000942 1778355309.528920 mapread 0x00824a thru 0x0157a2 (0xd559 bytes) 000945 1778355309.535736 read 0x0014de thru 0x00ba54 (0xa577 bytes) 000947 1778355309.536466 mapwrite 0x00fe55 thru 0x0175f7 (0x77a3 bytes) 000949 1778355309.553850 trunc from 0x037b9d to 0x032b8d (0x500f bytes) 000950 1778355309.564899 trunc from 0x032b8e to 0x027022 (0xbb6b bytes) 000951 1778355309.571269 MIRROR_RESYNC 5 mirrors 000952 1778355309.817320 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) 000953 1778355309.916441 trunc from 0x027023 to 0x016e7e (0x101a4 bytes) 000956 1778355309.946224 trunc from 0x016e7f to 0x00679c (0x106e2 bytes) 000960 1778355309.954923 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x679d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x679d): Mirror 2: 0xece7302a (stale) Mirror 3: 0xece7302a (stale) Mirror 4: 0xece7302a (stale) CRC-32 checksum value for chunk [0, 0x679d): Mirror 1: 0xece7302a Mirror 5: 0xece7302a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x679d: skipped 000961 1778355310.217475 trunc from 0x00679d to 0x02f309 (0x28b6d bytes) 000963 1778355310.243289 read 0x026344 thru 0x029029 (0x2ce6 bytes) 000964 1778355310.246182 mapread 0x01014e thru 0x019c14 (0x9ac7 bytes) 000965 1778355310.254237 mapread 0x00e9fb thru 0x010523 (0x1b29 bytes) 000966 1778355310.255162 trunc from 0x02f30a to 0x01c1e8 (0x13121 bytes) 000968 1778355310.266971 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1c1e9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c1e9): Mirror 2: 0xecc188d (stale) Mirror 3: 0xecc188d (stale) Mirror 4: 0xecc188d (stale) Mirror 5: 0xfa826682 (stale) CRC-32 checksum value for chunk [0, 0x1c1e9): Mirror 1: 0xfa826682 Mirror 6: 0xfa826682 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c1e9: skipped 000969 1778355310.526478 mapwrite 0x022944 thru 0x030e00 (0xe4bd bytes) 000970 1778355310.603821 write 0x022ae6 thru 0x028e58 (0x6373 bytes) 000971 1778355310.606005 read 0x029093 thru 0x02e39b (0x5309 bytes) 000972 1778355310.607169 trunc from 0x030e01 to 0x032b39 (0x1d39 bytes) 000973 1778355310.616505 write 0x03f216 thru 0x03ffff (0xdea bytes) 000976 1778355310.618438 read 0x00d9b4 thru 0x01c270 (0xe8bd bytes) 000977 1778355310.623039 mapread 0x0145ed thru 0x01b7bf (0x71d3 bytes) 000978 1778355310.626924 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x499306d2 (stale) Mirror 4: 0x499306d2 (stale) Mirror 5: 0xbb9f9a22 (stale) Mirror 6: 0xbb9f9a22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x499306d2 (stale) Mirror 4: 0x499306d2 (stale) Mirror 5: 0xbb9f9a22 (stale) Mirror 6: 0xbb9f9a22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000981 1778355310.774072 read 0x015963 thru 0x015e3a (0x4d8 bytes) 000982 1778355310.776409 read 0x007bd2 thru 0x0116ff (0x9b2e bytes) 000983 1778355310.781277 write 0x00661d thru 0x00abe8 (0x45cc bytes) 000987 1778355310.784100 write 0x00c110 thru 0x018484 (0xc375 bytes) 000988 1778355310.785301 mapwrite 0x03c91a thru 0x03ffff (0x36e6 bytes) 000990 1778355310.805094 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x499306d2 (stale) Mirror 4: 0x499306d2 (stale) Mirror 5: 0xbb9f9a22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x499306d2 (stale) Mirror 4: 0x499306d2 (stale) Mirror 5: 0xbb9f9a22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000991 1778355310.922084 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x499306d2 (stale) Mirror 5: 0xbb9f9a22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x499306d2 (stale) Mirror 5: 0xbb9f9a22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000992 1778355311.027468 read 0x013dd8 thru 0x022b36 (0xed5f bytes) 000994 1778355311.033713 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x5c0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbb9f9a22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbb9f9a22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000995 1778355311.119805 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xbb9f9a22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5ab8e2a4 Mirror 6: 0x5ab8e2a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000996 1778355311.394966 MIRROR_RESYNC 3 mirrors 000997 1778355311.631610 MIRROR_EXTEND to 4 mirrors 001000 1778355311.878550 read 0x016f30 thru 0x019650 (0x2721 bytes) All operations completed A-OK! PASS 70a (28s) == sanity-flr test 71: check mirror extend parent fid ==== 15:35:13 (1778355313) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (2s) == sanity-flr test 72: check append on FLR file ========== 15:35:15 (1778355315) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x88b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x88f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106896 s, 19.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x88b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x88c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x88f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602218 s, 17.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x88b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x88c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x890:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x88f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x88b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x88c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x890:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x88f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x88d:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (4s) == sanity-flr test 200a: stress test ===================== 15:35:19 (1778355319) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8401952 .. Extending file size to 6055776 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6609120 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7928160 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7947488 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8557472 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9354944 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9023040 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9209312 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9274624 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 59044 59045 59047 59048 59051 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) 192.168.203.101@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:36:24 (1778355384) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7081632 .. Extending file size to 2491232 .. Extending file size to 8311840 .. Extending file size to 6062304 .. done Extending file size to 6737024 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7008736 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7535104 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9148288 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8632960 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 9122752 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62377 62378 62380 62382 62384 done 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) 192.168.203.101@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (66s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:37:30 (1778355450) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 3584 2204672 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0337799 s, 62.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0890797 s, 23.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6edee1b4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:37:36 (1778355456) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0319116 s, 65.7 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 15:37:38 (1778355458) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x60e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8a6:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8a6:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8a7:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x8a2:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:37:41 (1778355461) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0402073 s, 52.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8a3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8a4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:37:44 (1778355464) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.05247 s, 40.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8aa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8a5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8aa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8a5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8a6:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:37:47 (1778355467) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0875084 s, 24.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8a7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8a7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8a8:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:38:11 (1778355491) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351782 s, 29.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ac:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8a9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ac:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8a9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ad:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:38:56 (1778355536) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ac:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0444368 s, 47.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ae:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:39:42 (1778355582) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg301-server mds-ost sync done. Creating new pool test_204f oleg301-server: Pool lustre.test_204f created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg301-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0448647 s, 46.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x8ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8af:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg301-server: Pool lustre.test_204f destroyed PASS 204f (54s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:40:36 (1778355636) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:40:38 (1778355638) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:40:41 (1778355641) oleg301-server: Pool lustre.test_206 created Adding targets to pool oleg301-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg301-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg301-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg301-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg301-server: Pool lustre.test_206 destroyed PASS 206 (16s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:40:57 (1778355657) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285171 s, 36.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011293 s, 92.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1e897ddc Mirror 3: 0x1e897ddc PASS 207 (4s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:41:01 (1778355661) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:41:03 (1778355663) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:41:04 (1778355664) Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0292845 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x19c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1580000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1580000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1580000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c20000 Read this (4 bytes): BBBB 192.168.203.101@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg301-client.virtnet /mnt/lustre2 (opts:) PASS 209a (18s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:41:22 (1778355682) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00318405 s, 2.6 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:41:24 (1778355684) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016062 s, 65.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x633 lmm_fid: [0x200000404:0x633:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x633 lmm_fid: [0x200000404:0x633:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x8ba:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835507 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-09 15:41:25.000000000 -0400 Modify: 2026-05-09 15:41:25.000000000 -0400 Change: 2026-05-09 15:41:25.000000000 -0400 Birth: 2026-05-09 15:41:25.000000000 -0400 PASS 210a (3s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:41:27 (1778355687) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (1s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:41:28 (1778355688) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.12194 s, 336 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0159282 s, 257 kB/s size after second write -rw-r--r-- 1 root root 4096 May 9 15:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x635:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 9 15:41 /mnt/lustre/f211.sanity-flr PASS 211 (3s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:41:31 (1778355691) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:41:33 (1778355693) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 812 sec ============ 15:41:36 (1778355696) === sanity-flr: start cleanup 15:41:37 (1778355697) === === sanity-flr: finish cleanup 15:41:39 (1778355699) ===