-----============= acceptance-small: sanity-flr ============----- Thu Jul 16 03:06:15 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg244-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 03:06:35 (1784185595) === oleg244-client.virtnet: executing check_config_client /mnt/lustre oleg244-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg244-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8deb86468000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8deb86468000.idle_timeout=debug disable quota as required oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 03:07:09 (1784185629) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7601] == sanity-flr test 0a: lfs mirror create with -N option == 03:07:11 (1784185631) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:07:27 (1784185647) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:07:32 (1784185652) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:07:36 (1784185656) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:08:02 (1784185682) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:08:05 (1784185685) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 03:08:10 (1784185690) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8deb86468000.stats=clear osc.lustre-OST0001-osc-ffff8deb86468000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:08:18 (1784185698) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:08:31 (1784185711) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:08:40 (1784185720) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 03:08:44 (1784185724) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:08:51 (1784185731) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 03:08:59 (1784185739) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:09:06 (1784185746) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:09:10 (1784185750) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:09:17 (1784185757) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:09:27 (1784185767) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:09:33 (1784185773) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.85157 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129357 s, 8.1 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:09:44 (1784185784) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.231524 s, 31.7 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 03:09:52 (1784185792) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:10:03 (1784185803) Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8deb86468000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8deb86468000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8deb86468000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8deb86468000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg244-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:11:19 (1784185879) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.324201 s, 22.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8deb86468000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8deb86468000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:12:06 (1784185926) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8deb86468000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8deb86468000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8deb86468000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8deb86468000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg244-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:13:32 (1784186012) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.19554 s, 12.5 MB/s -rw-r--r-- 1 root root 52428800 Jul 16 03:13 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.071s user 0m0.000s sys 0m0.179s Stopping /mnt/lustre-ost1 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8deb86468000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8deb86468000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.281s user 0m0.001s sys 0m0.213s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg244-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg244-server oleg244-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8deb86468000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8deb86468000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.305s user 0m0.001s sys 0m0.198s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg244-server: oleg244-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg244-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg244-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8deb86468000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8deb86468000.ost_server_uuid in FULL state after 0 sec PASS 33b (111s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:15:23 (1784186123) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:15:27 (1784186127) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:15:32 (1784186132) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 03:15:35 (1784186135) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 03:15:44 (1784186144) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.764664 s, 16.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:15:57 (1784186157) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534055 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606389 s, 17.3 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:16:06 (1784186166) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:16:16 (1784186176) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00120432 s, 850 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0014778 s, 693 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00151316 s, 677 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010429 s, 982 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 03:16:27 (1784186187) create test files with size 15730002 .. 5a3c383a3adec1c5889a1ffeee1eed33 - 56b0b7de6b46c910d458f0ce217330e2 - 48bad9c9e71202dc7af4c76874b1aee2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg244-client.virtnet /mnt/lustre (opts:) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 03:16:57 (1784186217) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (20s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:17:17 (1784186237) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:17:25 (1784186245) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210109 s, 5.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251621 s, 41.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 03:17:36 (1784186256) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0888274 s, 23.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.289058 s, 14.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166099 s, 12.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 03:17:52 (1784186272) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 03:17:56 (1784186276) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:18:00 (1784186280) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376547 s, 27.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 03:18:08 (1784186288) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:18:12 (1784186292) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:18:23 (1784186303) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.306914 s, 34.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg244-client.virtnet /mnt/lustre (opts:) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:18:34 (1784186314) llite.lustre-ffff8debb4ece000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.380377 s, 27.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg244-client.virtnet /mnt/lustre (opts:) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8deb98686000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:18:50 (1784186330) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:19:37 (1784186377) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (48s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:20:25 (1784186425) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 03:20:29 (1784186429) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.305195 s, 10.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.270322 s, 11.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa3f4fae1 (nosync,stale) Mirror 2: 0xa3f4fae1 (nosync,stale) ** verify mirror 2 doesn't change original checksum: f633f575e6d540fb3257125d9709ddc8 - mirror 1 checksum: d3a37fd3f8a8ccc737c4591b2de42622 - mirror 2 checksum: f633f575e6d540fb3257125d9709ddc8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-16T03:20:34-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-16T03:20:34-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f633f575e6d540fb3257125d9709ddc8 - mirror 1 checksum: d3a37fd3f8a8ccc737c4591b2de42622 - mirror 2 checksum: d3a37fd3f8a8ccc737c4591b2de42622 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-16T03:20:34-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-16T03:20:34-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (15s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:20:46 (1784186446) Failing mds1 on oleg244-server Stopping /mnt/lustre-mds1 (opts:) on oleg244-server 03:21:01 (1784186461) shut down facet: mds1 facet_host: oleg244-server facet_failover_host: oleg244-server Failover mds1 to oleg244-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 03:21:27 (1784186487) targets are mounted 03:21:27 (1784186487) facet_failover done oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290542 s, 36.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (65s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:21:51 (1784186511) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:21:55 (1784186515) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:21:59 (1784186519) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0111035 s, 1.5 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:22:09 (1784186529) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:22:13 (1784186533) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:22:17 (1784186537) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:22:45 (1784186565) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:23:16 (1784186596) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:23:41 (1784186621) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:24:06 (1784186646) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 03:24:14 (1784186654) mirror create pid 53940 mirror split pid 53941 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6943: 53941 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6943: 53941 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53940 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53941 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 03:25:22 (1784186722) Seed set to 6725 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1784186725.475428 write 0x02f144 thru 0x03844f (0x930c bytes) 000003 1784186725.498896 mapread 0x019e4a thru 0x029c59 (0xfe10 bytes) 000004 1784186725.559071 mapread 0x03754c thru 0x03844f (0xf04 bytes) 000006 1784186725.571272 mapread 0x009979 thru 0x010e02 (0x748a bytes) 000008 1784186725.602967 read 0x016d89 thru 0x024f87 (0xe1ff bytes) 000011 1784186725.608039 mapwrite 0x00b52a thru 0x00d539 (0x2010 bytes) 000012 1784186725.756308 mapwrite 0x0339d5 thru 0x0375c3 (0x3bef bytes) 000013 1784186725.866930 read 0x012371 thru 0x01d27a (0xaf0a bytes) 000014 1784186725.905893 mapwrite 0x01bf07 thru 0x01c3eb (0x4e5 bytes) 000015 1784186725.957650 read 0x03417e thru 0x03844f (0x42d2 bytes) 000017 1784186725.959854 write 0x00eaaa thru 0x018af9 (0xa050 bytes) 000018 1784186725.962120 mapread 0x01c76b thru 0x022079 (0x590f bytes) 000027 1784186725.971777 mapread 0x0212e0 thru 0x02fb64 (0xe885 bytes) 000028 1784186725.974139 write 0x02ac11 thru 0x03a736 (0xfb26 bytes) 000030 1784186725.978538 write 0x00a3d1 thru 0x019bc3 (0xf7f3 bytes) 000031 1784186725.981595 write 0x0365c1 thru 0x03c793 (0x61d3 bytes) 000035 1784186725.986108 mapread 0x0377db thru 0x03c793 (0x4fb9 bytes) 000038 1784186725.989760 mapwrite 0x03d602 thru 0x03ffff (0x29fe bytes) 000039 1784186726.097567 write 0x036c5e thru 0x03ffff (0x93a2 bytes) 000040 1784186726.103978 write 0x012c18 thru 0x01ad19 (0x8102 bytes) 000041 1784186726.108278 mapread 0x02ddfb thru 0x03d808 (0xfa0e bytes) 000042 1784186726.111798 write 0x00c11b thru 0x0187ef (0xc6d5 bytes) truncating to largest ever: 0x39242 000045 1784186726.119137 trunc from 0x040000 to 0x039241 (0x6dbe bytes) 000048 1784186726.160071 trunc from 0x039242 to 0x038f38 (0x309 bytes) 000049 1784186726.186317 trunc from 0x038f39 to 0x003fc8 (0x34f70 bytes) 000051 1784186726.212173 write 0x00706f thru 0x016ee4 (0xfe76 bytes) truncating to largest ever: 0x393d8 000054 1784186726.218565 trunc from 0x016ee5 to 0x0393d7 (0x224f3 bytes) 000055 1784186726.256208 mapread 0x0242ab thru 0x02cdb2 (0x8b08 bytes) 000056 1784186726.351019 read 0x034d20 thru 0x0393d7 (0x46b8 bytes) 000057 1784186726.369982 mapwrite 0x031fc4 thru 0x03ffff (0xe03c bytes) 000058 1784186726.531306 write 0x0374ef thru 0x03ffff (0x8b11 bytes) 000060 1784186726.553150 trunc from 0x040000 to 0x01c5ab (0x23a54 bytes) 000061 1784186726.600784 trunc from 0x01c5ac to 0x025fe0 (0x9a35 bytes) 000062 1784186726.623568 write 0x035403 thru 0x038253 (0x2e51 bytes) 000064 1784186726.626586 write 0x024947 thru 0x033a06 (0xf0c0 bytes) 000065 1784186726.639539 mapwrite 0x00c428 thru 0x00dc90 (0x1869 bytes) 000067 1784186726.720492 write 0x008ddd thru 0x00d772 (0x4996 bytes) 000069 1784186726.722402 mapread 0x016d2a thru 0x019fe5 (0x32bc bytes) 000070 1784186726.735310 trunc from 0x038254 to 0x034de1 (0x3472 bytes) 000072 1784186726.774121 trunc from 0x034de2 to 0x02e2c2 (0x6b1f bytes) 000073 1784186726.805982 MIRROR_EXTEND to 2 mirrors 000075 1784186727.944553 mapread 0x02b4ab thru 0x02e2c2 (0x2e18 bytes) 000076 1784186727.986182 mapwrite 0x008c1c thru 0x00d750 (0x4b35 bytes) 000077 1784186728.236393 mapwrite 0x01ff18 thru 0x02fc22 (0xfd0b bytes) 000079 1784186728.411676 mapwrite 0x030ed6 thru 0x03ec8b (0xddb6 bytes) 000081 1784186728.696398 mapwrite 0x031133 thru 0x031d72 (0xc40 bytes) 000082 1784186728.803556 read 0x02f181 thru 0x03316a (0x3fea bytes) 000083 1784186728.804397 read 0x03a23c thru 0x03e4af (0x4274 bytes) 000085 1784186728.813637 write 0x028cf7 thru 0x033d61 (0xb06b bytes) 000086 1784186728.815480 write 0x02df81 thru 0x03aa23 (0xcaa3 bytes) 000088 1784186728.823340 write 0x007f02 thru 0x00fa41 (0x7b40 bytes) 000089 1784186728.825668 mapread 0x03a2f8 thru 0x03ec8b (0x4994 bytes) 000091 1784186728.830390 write 0x01f4b0 thru 0x029c71 (0xa7c2 bytes) 000093 1784186728.835419 mapwrite 0x038557 thru 0x03ffff (0x7aa9 bytes) 000094 1784186729.015993 trunc from 0x040000 to 0x017071 (0x28f8e bytes) 000096 1784186729.042516 write 0x0119a6 thru 0x0165c7 (0x4c22 bytes) 000097 1784186729.044848 mapread 0x0133da thru 0x017071 (0x3c98 bytes) 000099 1784186729.046703 mapwrite 0x012c00 thru 0x020c2d (0xe02e bytes) 000100 1784186729.216631 trunc from 0x020c2e to 0x034d53 (0x14126 bytes) 000101 1784186729.261050 trunc from 0x034d54 to 0x029ce8 (0xb06b bytes) 000102 1784186729.309660 write 0x019391 thru 0x01a2a9 (0xf19 bytes) 000103 1784186729.313937 write 0x003ee2 thru 0x00b60b (0x772a bytes) 000104 1784186729.320644 mapread 0x006247 thru 0x014b2c (0xe8e6 bytes) 000108 1784186729.339217 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) 000109 1784186729.660797 mapread 0x023cd9 thru 0x02895a (0x4c82 bytes) 000110 1784186729.697271 write 0x01950e thru 0x01bb54 (0x2647 bytes) 000111 1784186729.717162 read 0x028fde thru 0x029ce8 (0xd0b bytes) 000113 1784186729.718565 write 0x03ead0 thru 0x03ffff (0x1530 bytes) 000114 1784186729.725702 MIRROR_EXTEND to 2 mirrors 000115 1784186730.688957 MIRROR_RESYNC 2 mirrors 000116 1784186731.270108 read 0x020b17 thru 0x02cb57 (0xc041 bytes) 000118 1784186731.293971 trunc from 0x040000 to 0x01c8d8 (0x23727 bytes) 000122 1784186731.497217 read 0x016afb thru 0x01a42f (0x3935 bytes) 000123 1784186731.512611 mapwrite 0x01bcd9 thru 0x02bb8e (0xfeb6 bytes) 000124 1784186731.632850 mapwrite 0x03a678 thru 0x03fdff (0x5788 bytes) 000125 1784186731.819626 trunc from 0x03fe00 to 0x011030 (0x2edcf bytes) 000126 1784186731.850830 mapread 0x00513f thru 0x00e6c3 (0x9585 bytes) 000128 1784186731.855006 read 0x0034fc thru 0x006237 (0x2d3c bytes) 000130 1784186731.857661 read 0x004c7c thru 0x00b1a8 (0x652d bytes) 000131 1784186731.858790 mapread 0x000fac thru 0x00d923 (0xc978 bytes) 000133 1784186731.862521 write 0x0052ed thru 0x00e1ad (0x8ec1 bytes) 000136 1784186731.872353 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) 000137 1784186732.118854 mapread 0x0084e3 thru 0x00f1c3 (0x6ce1 bytes) 000139 1784186732.147775 trunc from 0x011031 to 0x01b38a (0xa35a bytes) 000140 1784186732.168838 mapread 0x01aa6b thru 0x01b38a (0x920 bytes) 000141 1784186732.191284 read 0x00ce92 thru 0x0167ab (0x991a bytes) 000142 1784186732.210093 write 0x005fd2 thru 0x011f0b (0xbf3a bytes) 000144 1784186732.220080 mapwrite 0x03cf9b thru 0x03ffff (0x3065 bytes) 000145 1784186732.399349 trunc from 0x040000 to 0x008551 (0x37aae bytes) truncating to largest ever: 0x3a85b 000146 1784186732.442315 trunc from 0x008552 to 0x03a85a (0x32309 bytes) 000148 1784186732.507801 trunc from 0x03a85b to 0x00a309 (0x30551 bytes) 000149 1784186732.548214 mapwrite 0x002d05 thru 0x003b04 (0xe00 bytes) 000151 1784186732.674411 mapread 0x004adb thru 0x0095cc (0x4af2 bytes) 000152 1784186732.675375 mapread 0x000ab2 thru 0x0067bf (0x5d0e bytes) 000153 1784186732.692437 trunc from 0x00a30a to 0x03058e (0x26285 bytes) 000154 1784186732.724826 trunc from 0x03058f to 0x02b466 (0x5128 bytes) 000155 1784186732.756803 mapread 0x01ea14 thru 0x02726e (0x885b bytes) 000156 1784186732.788153 mapwrite 0x036365 thru 0x03ff00 (0x9b9c bytes) 000157 1784186732.928259 mapwrite 0x01bfe4 thru 0x020568 (0x4585 bytes) 000160 1784186733.062875 trunc from 0x03ff01 to 0x008746 (0x377ba bytes) 000162 1784186733.095488 MIRROR_EXTEND to 2 mirrors 000163 1784186734.071566 write 0x001935 thru 0x00d4ca (0xbb96 bytes) 000165 1784186734.234185 write 0x00ae8e thru 0x01481a (0x998d bytes) 000169 1784186734.250426 read 0x00f9a0 thru 0x011b66 (0x21c7 bytes) 000172 1784186734.252342 mapread 0x005489 thru 0x00ac1a (0x5792 bytes) 000173 1784186734.259043 mapread 0x00bd1f thru 0x0126d5 (0x69b7 bytes) 000174 1784186734.263965 trunc from 0x01481b to 0x01bf41 (0x7727 bytes) 000178 1784186734.347530 mapwrite 0x004831 thru 0x00b553 (0x6d23 bytes) 000179 1784186734.435160 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) 000180 1784186734.625243 mapread 0x00de3f thru 0x019510 (0xb6d2 bytes) 000183 1784186734.660179 write 0x03a746 thru 0x03ffff (0x58ba bytes) 000184 1784186734.680211 write 0x03578c thru 0x03ffff (0xa874 bytes) 000185 1784186734.693149 read 0x011b9c thru 0x01ed67 (0xd1cc bytes) 000186 1784186734.701310 mapread 0x029fef thru 0x02b215 (0x1227 bytes) 000187 1784186734.713189 trunc from 0x040000 to 0x01e9e5 (0x2161a bytes) 000188 1784186734.743606 read 0x009beb thru 0x010027 (0x643d bytes) 000191 1784186734.747192 mapwrite 0x0282dc thru 0x028784 (0x4a9 bytes) 000194 1784186734.917736 mapwrite 0x03d065 thru 0x03f96a (0x2906 bytes) 000195 1784186735.157566 write 0x00fcdb thru 0x017533 (0x7859 bytes) 000196 1784186735.159942 trunc from 0x03f96b to 0x036cc1 (0x8ca9 bytes) 000197 1784186735.189419 mapwrite 0x03c0f0 thru 0x03ffff (0x3f10 bytes) 000199 1784186735.352669 write 0x01514d thru 0x01801e (0x2ed2 bytes) 000200 1784186735.353845 mapwrite 0x022649 thru 0x02e1a8 (0xbb60 bytes) 000203 1784186735.544919 mapread 0x024283 thru 0x02fe3e (0xbbbc bytes) 000207 1784186735.572981 trunc from 0x040000 to 0x012dbe (0x2d241 bytes) 000208 1784186735.632998 mapwrite 0x007636 thru 0x015198 (0xdb63 bytes) 000209 1784186735.813793 mapread 0x00e402 thru 0x014906 (0x6505 bytes) 000210 1784186735.814938 write 0x03c83b thru 0x03ecaa (0x2470 bytes) 000212 1784186735.820297 trunc from 0x03ecab to 0x0346fa (0xa5b0 bytes) 000213 1784186735.852790 MIRROR_EXTEND to 2 mirrors 000217 1784186736.823044 MIRROR_EXTEND to 3 mirrors 000219 1784186737.943844 mapread 0x00cc98 thru 0x013d8f (0x70f8 bytes) 000220 1784186738.030423 trunc from 0x0346fb to 0x02ce4a (0x78b0 bytes) 000221 1784186738.197820 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ce4b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce4b): Mirror 2: 0xf77da703 (stale) Mirror 3: 0xf77da703 (stale) CRC-32 checksum value for chunk [0, 0x2ce4b): Mirror 1: 0xf77da703 Mirror 4: 0xf77da703 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce4b: skipped 000225 1784186739.673686 write 0x0399bf thru 0x03ffff (0x6641 bytes) 000226 1784186739.794204 mapread 0x032b8e thru 0x034f7f (0x23f2 bytes) 000227 1784186739.817628 write 0x03ccaf thru 0x03ffff (0x3351 bytes) 000228 1784186739.819184 write 0x03aa8f thru 0x03ffff (0x5571 bytes) 000229 1784186739.821544 read 0x037b9b thru 0x03ffff (0x8465 bytes) 000230 1784186739.826970 mapwrite 0x01c7fd thru 0x01e44d (0x1c51 bytes) 000231 1784186739.887277 write 0x020705 thru 0x02f9d5 (0xf2d1 bytes) 000233 1784186739.896545 mapwrite 0x00f562 thru 0x012aba (0x3559 bytes) 000234 1784186739.958051 read 0x013598 thru 0x02157e (0xdfe7 bytes) 000235 1784186739.959964 read 0x02e88e thru 0x03d7fe (0xef71 bytes) 000236 1784186739.961763 trunc from 0x040000 to 0x02b965 (0x1469a bytes) 000238 1784186739.990645 mapwrite 0x0184a3 thru 0x01cc23 (0x4781 bytes) 000239 1784186740.035540 mapread 0x00edb4 thru 0x0166f2 (0x793f bytes) 000243 1784186740.044877 MIRROR_RESYNC 4 mirrors 000246 1784186741.405403 read 0x0139a8 thru 0x01d0f8 (0x9751 bytes) 000250 1784186741.438596 trunc from 0x02b966 to 0x039636 (0xdcd1 bytes) 000255 1784186741.574661 mapwrite 0x00403b thru 0x007c05 (0x3bcb bytes) 000256 1784186741.706498 trunc from 0x039637 to 0x019fbf (0x1f677 bytes) 000257 1784186741.766692 read 0x018d75 thru 0x019fbf (0x124b bytes) 000259 1784186741.783357 read 0x00c0fa thru 0x011d99 (0x5ca0 bytes) 000260 1784186741.784160 mapwrite 0x033ca7 thru 0x039b47 (0x5ea1 bytes) 000261 1784186741.968133 MIRROR_RESYNC 4 mirrors 000265 1784186743.457698 trunc from 0x039b48 to 0x002e9e (0x36ca9 bytes) 000266 1784186743.548191 read 0x0008a6 thru 0x002e9e (0x25f9 bytes) 000268 1784186743.558694 mapwrite 0x01972c thru 0x01ece7 (0x55bc bytes) 000269 1784186743.758606 read 0x01e15f thru 0x01ece7 (0xb89 bytes) 000270 1784186743.760506 trunc from 0x01ece8 to 0x01e8c2 (0x425 bytes) 000273 1784186743.818987 read 0x00648a thru 0x012131 (0xbca8 bytes) 000274 1784186743.832814 mapwrite 0x003b8c thru 0x00628b (0x2700 bytes) 000277 1784186743.920204 read 0x00c071 thru 0x0179ba (0xb94a bytes) 000278 1784186743.932602 MIRROR_RESYNC 4 mirrors 000279 1784186745.021950 read 0x016595 thru 0x01e8c2 (0x832e bytes) 000282 1784186745.033884 mapwrite 0x00044b thru 0x004b4d (0x4703 bytes) 000283 1784186745.297200 trunc from 0x01e8c3 to 0x035171 (0x168af bytes) 000285 1784186745.328208 write 0x02e544 thru 0x037934 (0x93f1 bytes) 000286 1784186745.335667 mapwrite 0x0326f0 thru 0x03bf9f (0x98b0 bytes) 000287 1784186745.416851 write 0x03a465 thru 0x03af9d (0xb39 bytes) 000290 1784186745.420885 trunc from 0x03bfa0 to 0x036c27 (0x5378 bytes) 000291 1784186745.442372 mapwrite 0x0017c1 thru 0x006650 (0x4e90 bytes) 000292 1784186745.507390 trunc from 0x036c28 to 0x0250ab (0x11b7c bytes) 000293 1784186745.525072 mapread 0x0132bf thru 0x01ffe3 (0xcd25 bytes) 000297 1784186745.550749 MIRROR_RESYNC 4 mirrors 000299 1784186746.526872 mapread 0x017577 thru 0x01899c (0x1426 bytes) 000300 1784186746.541326 write 0x013141 thru 0x018cad (0x5b6d bytes) 000301 1784186746.617959 mapwrite 0x033a61 thru 0x03ffff (0xc59f bytes) 000303 1784186746.717209 write 0x00d68d thru 0x00f623 (0x1f97 bytes) 000305 1784186746.730082 mapread 0x00bfb8 thru 0x014d23 (0x8d6c bytes) 000308 1784186746.745163 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2f8f7b8e (stale) Mirror 4: 0x2f8f7b8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2f8f7b8e (stale) Mirror 4: 0x2f8f7b8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000309 1784186747.079170 read 0x023d5d thru 0x02822b (0x44cf bytes) 000311 1784186747.095512 mapwrite 0x01cdbd thru 0x023c0d (0x6e51 bytes) 000312 1784186747.214035 read 0x031885 thru 0x035c04 (0x4380 bytes) 000314 1784186747.249660 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2f8f7b8e (stale) Mirror 4: 0x2f8f7b8e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x937b3db9 Mirror 5: 0x937b3db9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000318 1784186748.539572 read 0x008f70 thru 0x00d5c0 (0x4651 bytes) 000323 1784186748.627372 mapread 0x0086f5 thru 0x010a38 (0x8344 bytes) 000324 1784186748.688089 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x2f8f7b8e (stale) Mirror 4: 0x2f8f7b8e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x937b3db9 Mirror 5: 0x937b3db9 Mirror 6: 0x937b3db9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000325 1784186750.025349 read 0x00554c thru 0x01342c (0xdee1 bytes) 000327 1784186750.060059 MIRROR_RESYNC 5 mirrors 000329 1784186751.374096 read 0x038c2d thru 0x03ffff (0x73d3 bytes) 000332 1784186751.388469 write 0x0062d5 thru 0x00eaa9 (0x87d5 bytes) 000333 1784186751.472583 read 0x039885 thru 0x03dcd5 (0x4451 bytes) 000335 1784186751.487874 trunc from 0x040000 to 0x00de82 (0x3217d bytes) 000336 1784186751.540431 mapwrite 0x026bb2 thru 0x035e3c (0xf28b bytes) 000338 1784186751.776857 read 0x0345ba thru 0x03472c (0x173 bytes) 000339 1784186751.780417 read 0x01055c thru 0x01c3a5 (0xbe4a bytes) 000340 1784186751.819094 mapwrite 0x0000f5 thru 0x00c55d (0xc469 bytes) 000342 1784186751.938625 mapwrite 0x0171db thru 0x01eab8 (0x78de bytes) 000343 1784186752.006055 write 0x00e75c thru 0x010a53 (0x22f8 bytes) 000345 1784186752.012769 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35e3d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e3d): Mirror 2: 0x6a10cc52 (stale) Mirror 4: 0x6a10cc52 (stale) Mirror 5: 0x6a10cc52 (stale) Mirror 6: 0x6a10cc52 (stale) CRC-32 checksum value for chunk [0, 0x35e3d): Mirror 1: 0xf8a4da1a Mirror 7: 0xf8a4da1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e3d: skipped 000346 1784186753.246947 write 0x0005c9 thru 0x003693 (0x30cb bytes) 000348 1784186753.358576 write 0x008300 thru 0x009a44 (0x1745 bytes) 000351 1784186753.390300 read 0x00c777 thru 0x0138f2 (0x717c bytes) 000353 1784186753.413423 mapwrite 0x003444 thru 0x009d45 (0x6902 bytes) 000354 1784186753.477165 mapread 0x019be1 thru 0x01fb22 (0x5f42 bytes) 000356 1784186753.493035 read 0x00f078 thru 0x01048e (0x1417 bytes) 000358 1784186753.495377 trunc from 0x035e3d to 0x02f49a (0x69a2 bytes) 000360 1784186753.518738 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f49b): Mirror 2: 0xbec6bda9 (stale) Mirror 4: 0xbec6bda9 (stale) Mirror 5: 0xbec6bda9 (stale) Mirror 6: 0xbec6bda9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f49b): Mirror 2: 0xbec6bda9 (stale) Mirror 4: 0xbec6bda9 (stale) Mirror 5: 0xbec6bda9 (stale) Mirror 6: 0xbec6bda9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f49b: skipped 000361 1784186753.900006 write 0x0370e4 thru 0x03ffff (0x8f1c bytes) 000362 1784186753.901459 read 0x016492 thru 0x021a59 (0xb5c8 bytes) 000363 1784186753.915456 mapread 0x0042a7 thru 0x00f1e1 (0xaf3b bytes) 000364 1784186753.947937 MIRROR_RESYNC 5 mirrors 000365 1784186755.450314 mapread 0x037940 thru 0x03ab9e (0x325f bytes) 000366 1784186755.485493 mapwrite 0x00d2ac thru 0x00f42d (0x2182 bytes) 000367 1784186755.727888 read 0x032ac5 thru 0x03b870 (0x8dac bytes) 000369 1784186755.738987 mapwrite 0x028517 thru 0x034886 (0xc370 bytes) 000372 1784186755.868388 trunc from 0x040000 to 0x024bec (0x1b413 bytes) 000373 1784186755.925951 read 0x015362 thru 0x01fdec (0xaa8b bytes) 000375 1784186755.979641 trunc from 0x024bed to 0x01e81f (0x63cd bytes) 000377 1784186756.012993 write 0x02e987 thru 0x030bf8 (0x2272 bytes) 000380 1784186756.019801 read 0x01f914 thru 0x027435 (0x7b22 bytes) 000381 1784186756.043136 mapread 0x029b13 thru 0x02b2e7 (0x17d5 bytes) 000383 1784186756.066511 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x30bf9): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30bf9): Mirror 2: 0x14e1ea74 (stale) Mirror 4: 0x14e1ea74 (stale) Mirror 5: 0x14e1ea74 (stale) Mirror 6: 0x14e1ea74 (stale) CRC-32 checksum value for chunk [0, 0x30bf9): Mirror 1: 0x83853f86 Mirror 7: 0x83853f86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30bf9: skipped 000384 1784186757.241190 MIRROR_RESYNC 6 mirrors 000385 1784186758.882644 write 0x03ec2b thru 0x03fd4e (0x1124 bytes) 000387 1784186758.946468 mapread 0x02df28 thru 0x033281 (0x535a bytes) 000388 1784186758.978125 write 0x0218b1 thru 0x02cd4a (0xb49a bytes) 000389 1784186758.988267 mapwrite 0x029d28 thru 0x032582 (0x885b bytes) 000391 1784186759.102177 mapwrite 0x0019d3 thru 0x001e83 (0x4b1 bytes) 000394 1784186759.231732 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd4f): Mirror 4: 0x38ad08f2 (stale) Mirror 5: 0x38ad08f2 (stale) Mirror 6: 0x38ad08f2 (stale) Mirror 7: 0x38ad08f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd4f): Mirror 4: 0x38ad08f2 (stale) Mirror 5: 0x38ad08f2 (stale) Mirror 6: 0x38ad08f2 (stale) Mirror 7: 0x38ad08f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd4f: skipped 000396 1784186759.660486 mapread 0x001666 thru 0x00e934 (0xd2cf bytes) 000397 1784186759.692387 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd4f): Mirror 4: 0x38ad08f2 (stale) Mirror 5: 0x38ad08f2 (stale) Mirror 7: 0x38ad08f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd4f): Mirror 4: 0x38ad08f2 (stale) Mirror 5: 0x38ad08f2 (stale) Mirror 7: 0x38ad08f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd4f: skipped 000398 1784186760.080847 write 0x039954 thru 0x03ffff (0x66ac bytes) 000399 1784186760.087123 mapwrite 0x03f6d1 thru 0x03ffff (0x92f bytes) 000400 1784186760.136318 mapwrite 0x00187d thru 0x00e68c (0xce10 bytes) 000401 1784186760.201870 mapread 0x034c84 thru 0x038e06 (0x4183 bytes) 000403 1784186760.216350 write 0x004b65 thru 0x004e8a (0x326 bytes) 000407 1784186760.225362 MIRROR_RESYNC 4 mirrors 000409 1784186761.114542 trunc from 0x040000 to 0x028a1a (0x175e5 bytes) 000411 1784186761.232491 mapwrite 0x017d64 thru 0x02567e (0xd91b bytes) 000413 1784186761.328621 mapread 0x00b578 thru 0x00e5bd (0x3046 bytes) 000414 1784186761.345532 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28a1b): Mirror 5: 0xfe6b9697 (stale) Mirror 7: 0xfe6b9697 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28a1b): Mirror 5: 0xfe6b9697 (stale) Mirror 7: 0xfe6b9697 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28a1b: skipped 000418 1784186761.748919 write 0x03cd86 thru 0x03ffff (0x327a bytes) 000419 1784186761.762217 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe9e26d32 (stale) Mirror 7: 0xe9e26d32 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb650f3b4 Mirror 8: 0xb650f3b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000422 1784186762.506822 mapread 0x03a61d thru 0x03ffff (0x59e3 bytes) 000423 1784186762.544614 mapread 0x032cec thru 0x03ffff (0xd314 bytes) 000424 1784186762.569806 mapwrite 0x025051 thru 0x03216e (0xd11e bytes) 000429 1784186762.769965 mapread 0x014d1f thru 0x01b3ed (0x66cf bytes) 000430 1784186762.782893 write 0x02ca3f thru 0x038472 (0xba34 bytes) 000433 1784186762.790458 read 0x009ab2 thru 0x0176ed (0xdc3c bytes) 000435 1784186762.803064 trunc from 0x040000 to 0x02dfc9 (0x12036 bytes) 000438 1784186762.837442 mapread 0x0004d1 thru 0x002cd9 (0x2809 bytes) 000439 1784186762.856345 mapread 0x01cb35 thru 0x02acf7 (0xe1c3 bytes) 000440 1784186762.869218 read 0x01c9b5 thru 0x029b02 (0xd14e bytes) 000441 1784186762.869964 mapread 0x00cd33 thru 0x01c8be (0xfb8c bytes) 000442 1784186762.873968 mapwrite 0x026dc9 thru 0x02d34e (0x6586 bytes) 000443 1784186762.966908 mapwrite 0x035ec9 thru 0x0385cf (0x2707 bytes) 000445 1784186763.062531 trunc from 0x0385d0 to 0x03582c (0x2da3 bytes) 000446 1784186763.091526 mapread 0x01bb5c thru 0x022c3b (0x70e0 bytes) 000447 1784186763.098049 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3582d): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3582d): Mirror 5: 0x3c2f8b59 (stale) Mirror 7: 0x3c2f8b59 (stale) Mirror 8: 0x55030e53 (stale) CRC-32 checksum value for chunk [0, 0x3582d): Mirror 1: 0x4aed8eb Mirror 9: 0x4aed8eb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3582d: skipped 000450 1784186763.974085 MIRROR_RESYNC 5 mirrors 000452 1784186764.793996 write 0x021619 thru 0x0228bb (0x12a3 bytes) 000453 1784186764.875016 mapwrite 0x03c6c8 thru 0x03ffff (0x3938 bytes) 000454 1784186764.986150 mapwrite 0x02fbde thru 0x034c90 (0x50b3 bytes) 000456 1784186765.036535 mapwrite 0x026b20 thru 0x02b85b (0x4d3c bytes) 000461 1784186765.105543 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa4b14300 (stale) Mirror 8: 0xa4b14300 (stale) Mirror 9: 0xa4b14300 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa4b14300 (stale) Mirror 8: 0xa4b14300 (stale) Mirror 9: 0xa4b14300 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000462 1784186765.456934 mapread 0x03b995 thru 0x03bec0 (0x52c bytes) 000463 1784186765.468098 trunc from 0x040000 to 0x01fff2 (0x2000d bytes) 000465 1784186765.509895 read 0x0175fe thru 0x01fff2 (0x89f5 bytes) 000466 1784186765.525300 read 0x000a76 thru 0x00e9ec (0xdf77 bytes) 000467 1784186765.547599 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1fff3): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fff3): Mirror 7: 0x4758964c (stale) Mirror 8: 0x4758964c (stale) Mirror 9: 0x4758964c (stale) CRC-32 checksum value for chunk [0, 0x1fff3): Mirror 1: 0x4758964c Mirror 10: 0x4758964c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fff3: skipped 000468 1784186766.282809 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fff3): Mirror 7: 0x4758964c (stale) Mirror 8: 0x4758964c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fff3): Mirror 7: 0x4758964c (stale) Mirror 8: 0x4758964c (stale) CRC-32 checksum value for chunk [0, 0x1fff3): Mirror 1: 0x4758964c Mirror 10: 0x4758964c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fff3: skipped 000470 1784186766.748922 mapread 0x018b03 thru 0x01fff2 (0x74f0 bytes) 000472 1784186766.803783 MIRROR_RESYNC 4 mirrors 000473 1784186767.591946 mapread 0x0123e7 thru 0x01ae2d (0x8a47 bytes) 000476 1784186767.619402 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) 000477 1784186767.933730 trunc from 0x01fff3 to 0x026d6f (0x6d7d bytes) 000480 1784186768.049850 read 0x00e77b thru 0x016a3f (0x82c5 bytes) 000482 1784186768.060175 mapwrite 0x021cc6 thru 0x023013 (0x134e bytes) 000483 1784186768.131795 trunc from 0x026d70 to 0x03994a (0x12bdb bytes) 000485 1784186768.164625 mapread 0x027be4 thru 0x028f35 (0x1352 bytes) 000488 1784186768.179157 trunc from 0x03994b to 0x009e32 (0x2fb18 bytes) 000491 1784186768.196139 mapread 0x001686 thru 0x009e32 (0x87ad bytes) 000492 1784186768.211316 read 0x000772 thru 0x009e32 (0x96c1 bytes) 000493 1784186768.226311 MIRROR_RESYNC 3 mirrors 000494 1784186768.814465 write 0x00f909 thru 0x00ff7a (0x672 bytes) truncating to largest ever: 0x3cb71 000495 1784186768.873004 trunc from 0x00ff7b to 0x03cb70 (0x2cbf6 bytes) 000501 1784186768.918393 write 0x012235 thru 0x01e23c (0xc008 bytes) 000502 1784186768.929314 mapwrite 0x01890f thru 0x0240f4 (0xb7e6 bytes) 000504 1784186768.999371 read 0x02956c thru 0x038c08 (0xf69d bytes) truncating to largest ever: 0x3fbdf 000507 1784186769.015498 trunc from 0x03cb71 to 0x03fbde (0x306e bytes) 000508 1784186769.035976 read 0x021e1f thru 0x02b39f (0x9581 bytes) 000509 1784186769.037286 mapread 0x00c7d3 thru 0x011b50 (0x537e bytes) 000512 1784186769.048171 trunc from 0x03fbdf to 0x03c085 (0x3b59 bytes) 000517 1784186769.097413 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c086): Mirror 10: 0x51808130 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c086): Mirror 10: 0x51808130 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c086: skipped 000519 1784186769.458013 trunc from 0x03c086 to 0x03a81a (0x186b bytes) 000521 1784186769.485029 MIRROR_RESYNC 2 mirrors 000522 1784186770.152358 write 0x009e4a thru 0x0136d1 (0x9888 bytes) 000523 1784186770.246214 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) 000524 1784186770.389960 trunc from 0x03a81b to 0x0291b9 (0x11661 bytes) 000525 1784186770.436024 mapwrite 0x02c3e9 thru 0x02db4f (0x1767 bytes) 000530 1784186770.512828 mapread 0x00f75c thru 0x01548f (0x5d34 bytes) 000532 1784186770.522554 write 0x00a48f thru 0x00e0e7 (0x3c59 bytes) 000533 1784186770.523664 mapread 0x010af2 thru 0x013909 (0x2e18 bytes) 000534 1784186770.525399 write 0x000ef4 thru 0x00629f (0x53ac bytes) 000536 1784186770.536033 write 0x02fd1f thru 0x03f61d (0xf8ff bytes) 000539 1784186770.541448 trunc from 0x03f61e to 0x02868b (0x16f92 bytes) 000540 1784186770.559488 mapwrite 0x0154f6 thru 0x01dd49 (0x8854 bytes) 000541 1784186770.615648 mapwrite 0x0090d5 thru 0x00d521 (0x444d bytes) 000544 1784186770.663845 read 0x010c9c thru 0x01b7d4 (0xab39 bytes) 000547 1784186770.666668 write 0x030c6f thru 0x032af8 (0x1e8a bytes) 000551 1784186770.671886 write 0x02aad6 thru 0x039144 (0xe66f bytes) 000552 1784186770.677783 mapwrite 0x024dd9 thru 0x032de6 (0xe00e bytes) 000556 1784186770.740806 read 0x001978 thru 0x008fdd (0x7666 bytes) 000558 1784186770.748357 mapwrite 0x034e8b thru 0x036b46 (0x1cbc bytes) 000559 1784186770.789859 mapwrite 0x002e1d thru 0x006d93 (0x3f77 bytes) 000561 1784186770.837132 trunc from 0x039145 to 0x01a50d (0x1ec37 bytes) 000562 1784186770.851419 write 0x01ab6a thru 0x020a68 (0x5eff bytes) 000564 1784186770.853844 mapwrite 0x008be7 thru 0x00c94a (0x3d64 bytes) 000565 1784186770.898021 trunc from 0x020a69 to 0x008872 (0x181f6 bytes) 000567 1784186770.921382 trunc from 0x008873 to 0x039daf (0x3153d bytes) 000568 1784186770.939489 mapwrite 0x00d2e8 thru 0x00f7ae (0x24c7 bytes) 000569 1784186770.990615 trunc from 0x039db0 to 0x02a5c7 (0xf7e8 bytes) 000571 1784186771.018196 mapread 0x027898 thru 0x02a5c7 (0x2d30 bytes) 000573 1784186771.039672 mapwrite 0x01454a thru 0x023af4 (0xf5ab bytes) 000574 1784186771.334919 write 0x00da15 thru 0x00f7c5 (0x1db1 bytes) 000575 1784186771.338081 mapwrite 0x01c7d7 thru 0x024b1e (0x8348 bytes) 000577 1784186771.444025 mapwrite 0x02c1d6 thru 0x039449 (0xd274 bytes) 000579 1784186771.719548 mapwrite 0x005b23 thru 0x00fd4f (0xa22d bytes) 000580 1784186771.852319 write 0x019a26 thru 0x029548 (0xfb23 bytes) 000581 1784186771.858837 trunc from 0x03944a to 0x01fa5a (0x199ef bytes) 000582 1784186771.885846 mapread 0x0196d6 thru 0x01fa5a (0x6385 bytes) 000583 1784186771.891839 mapread 0x00a37e thru 0x00be2a (0x1aad bytes) 000584 1784186771.893954 trunc from 0x01fa5b to 0x00d80d (0x1224d bytes) 000589 1784186771.942593 write 0x02bfd9 thru 0x03871a (0xc742 bytes) 000590 1784186771.945819 mapread 0x01972b thru 0x026d95 (0xd66b bytes) 000592 1784186772.054772 read 0x0099e9 thru 0x012898 (0x8eb0 bytes) 000593 1784186772.066948 write 0x016471 thru 0x019ddb (0x396b bytes) 000594 1784186772.071451 mapread 0x00f63d thru 0x0159dc (0x63a0 bytes) 000595 1784186772.074566 mapread 0x011339 thru 0x011c55 (0x91d bytes) 000597 1784186772.077669 mapwrite 0x00c489 thru 0x017321 (0xae99 bytes) 000599 1784186772.173464 mapwrite 0x021d2a thru 0x02812d (0x6404 bytes) 000600 1784186772.347715 write 0x001305 thru 0x0063da (0x50d6 bytes) 000602 1784186772.363903 write 0x03b6de thru 0x03ffff (0x4922 bytes) 000606 1784186772.376890 mapwrite 0x02bf3f thru 0x031574 (0x5636 bytes) 000608 1784186772.544539 write 0x002054 thru 0x00ee20 (0xcdcd bytes) 000610 1784186772.556472 mapwrite 0x00a48b thru 0x011b5e (0x76d4 bytes) 000611 1784186772.649772 mapread 0x02d4a9 thru 0x037f92 (0xaaea bytes) 000612 1784186772.652418 mapread 0x013fc2 thru 0x021b26 (0xdb65 bytes) 000613 1784186772.655650 mapread 0x022341 thru 0x023d64 (0x1a24 bytes) 000614 1784186772.659937 mapread 0x029b86 thru 0x0355fc (0xba77 bytes) 000619 1784186772.670747 read 0x0229a1 thru 0x031844 (0xeea4 bytes) 000620 1784186772.671695 mapwrite 0x024c31 thru 0x0259d0 (0xda0 bytes) 000623 1784186772.797458 read 0x01bb7d thru 0x02bae3 (0xff67 bytes) 000624 1784186772.798435 write 0x010521 thru 0x01c68a (0xc16a bytes) 000628 1784186772.802859 mapwrite 0x02484e thru 0x02ab23 (0x62d6 bytes) 000631 1784186772.871348 mapread 0x03f8f1 thru 0x03ffff (0x70f bytes) 000632 1784186772.875811 write 0x033c99 thru 0x034b6b (0xed3 bytes) 000636 1784186772.883635 trunc from 0x040000 to 0x0143be (0x2bc41 bytes) 000637 1784186772.919729 mapwrite 0x023e28 thru 0x02cda8 (0x8f81 bytes) 000639 1784186773.082764 MIRROR_EXTEND to 2 mirrors 000640 1784186773.964036 write 0x03810b thru 0x03ffff (0x7ef5 bytes) 000643 1784186774.083929 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9b8a6c23 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x805dac90 Mirror 3: 0x805dac90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000644 1784186775.574723 read 0x00d819 thru 0x01cece (0xf6b6 bytes) 000645 1784186775.611886 trunc from 0x040000 to 0x01b017 (0x24fe8 bytes) 000646 1784186775.740378 mapwrite 0x00dd6e thru 0x00fc0d (0x1ea0 bytes) 000647 1784186775.869730 write 0x025d86 thru 0x02f6a0 (0x991b bytes) 000648 1784186775.872392 read 0x009a8e thru 0x00ffd1 (0x6544 bytes) 000651 1784186775.879721 read 0x008a82 thru 0x01872e (0xfcad bytes) 000652 1784186775.891388 mapwrite 0x00f252 thru 0x0150dd (0x5e8c bytes) 000653 1784186776.064977 mapread 0x028327 thru 0x02f6a0 (0x737a bytes) 000655 1784186776.066790 mapread 0x00975d thru 0x017216 (0xdaba bytes) 000656 1784186776.073304 read 0x012850 thru 0x01fca7 (0xd458 bytes) 000657 1784186776.094141 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f6a1): Mirror 2: 0x47f88b15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f6a1): Mirror 2: 0x47f88b15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f6a1: skipped 000658 1784186776.553470 read 0x0016da thru 0x00e609 (0xcf30 bytes) 000659 1784186776.565297 write 0x020c21 thru 0x025d35 (0x5115 bytes) 000660 1784186776.575930 mapread 0x0080fe thru 0x009131 (0x1034 bytes) 000663 1784186776.580375 mapwrite 0x03cc4e thru 0x03ffff (0x33b2 bytes) 000671 1784186776.682457 read 0x0205d6 thru 0x02b253 (0xac7e bytes) 000672 1784186776.699819 mapwrite 0x002163 thru 0x011a8b (0xf929 bytes) 000673 1784186776.779078 read 0x00401a thru 0x01012b (0xc112 bytes) 000675 1784186776.780459 read 0x013cfe thru 0x0228df (0xebe2 bytes) 000678 1784186776.787825 mapwrite 0x0108c7 thru 0x01936c (0x8aa6 bytes) 000679 1784186776.862408 mapwrite 0x00e9ba thru 0x01ce93 (0xe4da bytes) 000680 1784186776.925141 mapread 0x03c64e thru 0x03ffff (0x39b2 bytes) 000681 1784186776.926573 mapwrite 0x00ffd9 thru 0x01c9b9 (0xc9e1 bytes) 000682 1784186777.027330 trunc from 0x040000 to 0x0036bf (0x3c940 bytes) 000684 1784186777.062289 write 0x003f7f thru 0x00767f (0x3701 bytes) 000685 1784186777.066933 write 0x03a074 thru 0x03ffff (0x5f8c bytes) 000686 1784186777.068365 mapread 0x00d0d3 thru 0x0150d8 (0x8006 bytes) 000688 1784186777.080723 mapread 0x033d1f thru 0x03e871 (0xab53 bytes) 000690 1784186777.087641 mapread 0x0325af thru 0x036d7f (0x47d1 bytes) 000692 1784186777.090778 write 0x027706 thru 0x031317 (0x9c12 bytes) 000694 1784186777.093820 read 0x000456 thru 0x0073f8 (0x6fa3 bytes) 000695 1784186777.096076 trunc from 0x040000 to 0x023ff7 (0x1c008 bytes) 000697 1784186777.130369 MIRROR_RESYNC 2 mirrors 000698 1784186777.960180 mapread 0x003dce thru 0x0068ad (0x2ae0 bytes) 000702 1784186777.988552 mapread 0x014472 thru 0x01dc97 (0x9826 bytes) 000704 1784186778.011900 mapwrite 0x031d7a thru 0x03698b (0x4c12 bytes) 000706 1784186778.205083 trunc from 0x03698c to 0x03a0b6 (0x372b bytes) 000707 1784186778.233061 mapwrite 0x01529f thru 0x015897 (0x5f9 bytes) 000709 1784186778.313128 mapread 0x02b09b thru 0x02d003 (0x1f69 bytes) 000711 1784186778.315387 mapread 0x032828 thru 0x03a0b6 (0x788f bytes) 000712 1784186778.324402 write 0x01b0b5 thru 0x020199 (0x50e5 bytes) 000713 1784186778.330128 trunc from 0x03a0b7 to 0x03c274 (0x21be bytes) 000715 1784186778.355268 write 0x027b5c thru 0x02ec83 (0x7128 bytes) 000716 1784186778.370913 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) 000717 1784186778.562581 MIRROR_EXTEND to 2 mirrors 000723 1784186779.639325 read 0x037caf thru 0x03c274 (0x45c6 bytes) 000724 1784186779.666302 mapwrite 0x03511b thru 0x03e67a (0x9560 bytes) 000727 1784186779.910351 write 0x00e205 thru 0x012a2d (0x4829 bytes) 000729 1784186779.929404 trunc from 0x03e67b to 0x035bf4 (0x8a86 bytes) 000731 1784186779.958010 write 0x036a44 thru 0x03ffff (0x95bc bytes) 000732 1784186779.965129 MIRROR_RESYNC 2 mirrors 000736 1784186780.838649 MIRROR_EXTEND to 3 mirrors 000738 1784186781.806404 mapread 0x01e7dc thru 0x020dbc (0x25e1 bytes) 000741 1784186781.870917 MIRROR_RESYNC 3 mirrors 000742 1784186782.505029 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) 000745 1784186782.903342 trunc from 0x040000 to 0x02d3b5 (0x12c4a bytes) 000746 1784186783.028035 MIRROR_RESYNC 2 mirrors 000747 1784186783.867170 write 0x01a432 thru 0x02a28d (0xfe5c bytes) 000749 1784186784.010764 trunc from 0x02d3b6 to 0x00cdf7 (0x205be bytes) 000752 1784186784.079170 trunc from 0x00cdf8 to 0x010135 (0x333e bytes) 000753 1784186784.137671 trunc from 0x010136 to 0x0036dd (0xca58 bytes) 000755 1784186784.211802 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) 000757 1784186784.488200 mapread 0x0011e1 thru 0x0036dd (0x24fd bytes) 000758 1784186784.530299 trunc from 0x0036de to 0x01ca9c (0x193bf bytes) 000759 1784186784.560865 mapwrite 0x00b502 thru 0x018e16 (0xd915 bytes) 000762 1784186784.693588 mapwrite 0x02594f thru 0x02de5f (0x8511 bytes) 000764 1784186784.817039 write 0x039334 thru 0x03ffff (0x6ccc bytes) 000765 1784186784.820742 trunc from 0x040000 to 0x0314e9 (0xeb16 bytes) 000766 1784186784.842849 trunc from 0x0314ea to 0x00afdb (0x2650e bytes) 000767 1784186784.866866 read 0x0019e4 thru 0x00afdb (0x95f8 bytes) 000768 1784186784.869881 trunc from 0x00afdc to 0x027ef2 (0x1cf17 bytes) 000769 1784186784.893844 mapwrite 0x00ffe0 thru 0x01bf4e (0xbf6f bytes) 000774 1784186785.010022 mapwrite 0x012607 thru 0x01a305 (0x7cff bytes) 000775 1784186785.073628 write 0x0161ed thru 0x025bad (0xf9c1 bytes) 000776 1784186785.095181 read 0x0268d4 thru 0x027ef2 (0x161f bytes) 000781 1784186785.130579 MIRROR_EXTEND to 2 mirrors 000783 1784186786.313765 write 0x0102c0 thru 0x014a5b (0x479c bytes) 000785 1784186786.456826 mapread 0x01fd22 thru 0x027ef2 (0x81d1 bytes) 000788 1784186786.537362 write 0x02117f thru 0x02552f (0x43b1 bytes) 000789 1784186786.539824 write 0x02cc6c thru 0x031eb0 (0x5245 bytes) 000792 1784186786.556355 MIRROR_RESYNC 2 mirrors 000793 1784186787.634713 mapread 0x020161 thru 0x0248ef (0x478f bytes) 000794 1784186787.710269 trunc from 0x031eb1 to 0x03e8e4 (0xca34 bytes) 000797 1784186787.891291 mapread 0x02f2fd thru 0x03a73a (0xb43e bytes) 000799 1784186787.936273 mapwrite 0x03531e thru 0x03b9d3 (0x66b6 bytes) 000800 1784186788.062174 write 0x01c7e5 thru 0x02b3f8 (0xec14 bytes) 000802 1784186788.079642 write 0x0084ee thru 0x013b09 (0xb61c bytes) 000805 1784186788.103047 mapread 0x032b1a thru 0x0350a1 (0x2588 bytes) 000806 1784186788.105953 read 0x015cca thru 0x016c15 (0xf4c bytes) 000807 1784186788.115920 read 0x0078ca thru 0x008899 (0xfd0 bytes) 000808 1784186788.155287 write 0x010fec thru 0x01f814 (0xe829 bytes) 000810 1784186788.161535 mapread 0x002466 thru 0x00a5ad (0x8148 bytes) 000811 1784186788.163523 mapwrite 0x017d5e thru 0x01cf55 (0x51f8 bytes) 000812 1784186788.219791 mapread 0x036fb7 thru 0x03e8e4 (0x792e bytes) 000814 1784186788.221676 read 0x035eb1 thru 0x03e8e4 (0x8a34 bytes) 000820 1784186788.241346 read 0x00e7c5 thru 0x019396 (0xabd2 bytes) 000821 1784186788.242840 read 0x009326 thru 0x0187b7 (0xf492 bytes) 000823 1784186788.245206 write 0x00531b thru 0x0124ea (0xd1d0 bytes) 000826 1784186788.259637 trunc from 0x03e8e5 to 0x031cd0 (0xcc14 bytes) 000827 1784186788.305447 trunc from 0x031cd1 to 0x025e3d (0xbe93 bytes) 000829 1784186788.327798 mapread 0x0080aa thru 0x012e40 (0xad97 bytes) 000832 1784186788.331210 mapread 0x017920 thru 0x025c5e (0xe33f bytes) 000833 1784186788.333632 read 0x00b939 thru 0x00f399 (0x3a61 bytes) 000835 1784186788.335612 mapread 0x00caeb thru 0x01c4c1 (0xf9d7 bytes) 000836 1784186788.337716 mapwrite 0x007b26 thru 0x00a3e7 (0x28c2 bytes) 000837 1784186788.386013 trunc from 0x025e3e to 0x0220b1 (0x3d8c bytes) 000838 1784186788.403002 mapwrite 0x035ebe thru 0x03ffff (0xa142 bytes) 000839 1784186788.476125 write 0x034958 thru 0x03f516 (0xabbf bytes) 000840 1784186788.481625 read 0x035169 thru 0x03578b (0x623 bytes) 000841 1784186788.482959 mapread 0x02f033 thru 0x037f95 (0x8f63 bytes) 000843 1784186788.485481 mapwrite 0x0200cf thru 0x02c113 (0xc045 bytes) 000844 1784186788.543423 mapwrite 0x00dbe3 thru 0x0140b6 (0x64d4 bytes) 000845 1784186788.595283 trunc from 0x040000 to 0x011951 (0x2e6ae bytes) 000848 1784186788.631291 mapwrite 0x033474 thru 0x039670 (0x61fd bytes) 000849 1784186788.766652 mapwrite 0x033b75 thru 0x038386 (0x4812 bytes) 000854 1784186788.916502 mapwrite 0x03257c thru 0x03ffff (0xda84 bytes) 000856 1784186789.099765 trunc from 0x040000 to 0x01f49d (0x20b62 bytes) 000857 1784186789.134018 mapread 0x01d1e8 thru 0x01f49d (0x22b6 bytes) 000859 1784186789.137148 write 0x016e80 thru 0x01fe33 (0x8fb4 bytes) 000860 1784186789.140004 mapwrite 0x00c955 thru 0x01744f (0xaafb bytes) 000861 1784186789.234372 read 0x00e61e thru 0x01c45e (0xde41 bytes) 000862 1784186789.236026 trunc from 0x01fe34 to 0x00aaeb (0x15348 bytes) 000863 1784186789.289302 MIRROR_RESYNC 2 mirrors 000865 1784186790.225624 trunc from 0x00aaec to 0x00aefb (0x410 bytes) 000869 1784186790.315144 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xaefc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaefc): Mirror 2: 0x712dd0b4 (stale) CRC-32 checksum value for chunk [0, 0xaefc): Mirror 1: 0x712dd0b4 Mirror 3: 0x712dd0b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaefc: skipped 000870 1784186791.457603 write 0x01ea17 thru 0x028ee5 (0xa4cf bytes) 000871 1784186791.621367 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28ee6): Mirror 2: 0xc07a8124 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ee6): Mirror 2: 0xc07a8124 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ee6: skipped 000872 1784186792.185690 trunc from 0x028ee6 to 0x0179ef (0x114f6 bytes) 000874 1784186792.213929 write 0x029f1e thru 0x02b915 (0x19f8 bytes) 000875 1784186792.215249 write 0x030279 thru 0x038a26 (0x87ae bytes) 000877 1784186792.223814 read 0x037364 thru 0x03832f (0xfcc bytes) 000878 1784186792.224774 trunc from 0x038a27 to 0x0217d6 (0x17250 bytes) 000879 1784186792.244550 trunc from 0x0217d7 to 0x02d6ea (0xbf14 bytes) 000880 1784186792.279153 write 0x0253f7 thru 0x030547 (0xb151 bytes) 000881 1784186792.286837 mapwrite 0x0022e8 thru 0x004eb2 (0x2bcb bytes) 000882 1784186792.376528 trunc from 0x030548 to 0x008172 (0x283d5 bytes) 000883 1784186792.412527 write 0x018fa9 thru 0x020dbe (0x7e16 bytes) 000885 1784186792.424689 MIRROR_RESYNC 2 mirrors 000887 1784186793.471827 mapread 0x014b1c thru 0x01e07a (0x955f bytes) 000888 1784186793.506407 write 0x02f64f thru 0x032008 (0x29ba bytes) 000890 1784186793.675046 mapread 0x0101d3 thru 0x01d02e (0xce5c bytes) 000891 1784186793.728540 write 0x037851 thru 0x03afc3 (0x3773 bytes) 000892 1784186793.747568 MIRROR_RESYNC 2 mirrors 000895 1784186794.727191 MIRROR_EXTEND to 3 mirrors 000897 1784186795.614076 mapread 0x02fbf9 thru 0x03189c (0x1ca4 bytes) 000898 1784186795.643216 mapwrite 0x037951 thru 0x03d093 (0x5743 bytes) 000900 1784186795.790421 trunc from 0x03d094 to 0x02afdb (0x120b8 bytes) 000902 1784186795.838654 MIRROR_RESYNC 3 mirrors 000903 1784186796.820107 mapread 0x007da3 thru 0x0119e0 (0x9c3e bytes) 000905 1784186796.841070 write 0x018f4e thru 0x023969 (0xaa1c bytes) 000906 1784186796.906217 mapread 0x029a29 thru 0x02ae65 (0x143d bytes) 000907 1784186796.917568 write 0x0166d3 thru 0x0230d3 (0xca01 bytes) 000908 1784186796.924530 write 0x034b09 thru 0x03a5ba (0x5ab2 bytes) 000909 1784186796.926894 write 0x01a34d thru 0x01e08d (0x3d41 bytes) 000911 1784186796.934386 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a5bb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5bb): Mirror 2: 0x6dbc738 (stale) Mirror 3: 0x6dbc738 (stale) CRC-32 checksum value for chunk [0, 0x3a5bb): Mirror 1: 0x6918ee0d Mirror 4: 0x6918ee0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5bb: skipped 000913 1784186798.292578 read 0x036d92 thru 0x03a5ba (0x3829 bytes) 000915 1784186798.324753 mapread 0x039e56 thru 0x03a5ba (0x765 bytes) 000916 1784186798.354518 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a5bb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5bb): Mirror 2: 0x6dbc738 (stale) Mirror 3: 0x6dbc738 (stale) CRC-32 checksum value for chunk [0, 0x3a5bb): Mirror 1: 0x6918ee0d Mirror 4: 0x6918ee0d Mirror 5: 0x6918ee0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5bb: skipped 000917 1784186799.956011 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a5bb): Mirror 3: 0x6dbc738 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5bb): Mirror 3: 0x6dbc738 (stale) CRC-32 checksum value for chunk [0, 0x3a5bb): Mirror 1: 0x6918ee0d Mirror 4: 0x6918ee0d Mirror 5: 0x6918ee0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5bb: skipped 000918 1784186801.094053 write 0x00abb4 thru 0x01846d (0xd8ba bytes) 000919 1784186801.322652 write 0x026104 thru 0x031b23 (0xba20 bytes) 000922 1784186801.339577 mapread 0x0090f1 thru 0x013776 (0xa686 bytes) 000924 1784186801.352694 mapread 0x03940a thru 0x03a5ba (0x11b1 bytes) 000925 1784186801.372928 mapread 0x02e9fa thru 0x03a5ba (0xbbc1 bytes) 000929 1784186801.384470 trunc from 0x03a5bb to 0x011975 (0x28c45 bytes) 000932 1784186801.423937 read 0x00dc7f thru 0x011975 (0x3cf7 bytes) 000933 1784186801.425786 read 0x00666e thru 0x00eb77 (0x850a bytes) 000936 1784186801.441603 trunc from 0x011976 to 0x00965c (0x8319 bytes) 000939 1784186801.471599 trunc from 0x00965d to 0x016938 (0xd2dc bytes) 000940 1784186801.499486 read 0x00351f thru 0x00cef3 (0x99d5 bytes) 000941 1784186801.513199 mapwrite 0x00f640 thru 0x014a03 (0x53c4 bytes) 000942 1784186801.580938 read 0x00a651 thru 0x016938 (0xc2e8 bytes) 000943 1784186801.584569 mapwrite 0x02afb9 thru 0x039914 (0xe95c bytes) 000950 1784186801.736449 read 0x01d3a5 thru 0x02a9e8 (0xd644 bytes) 000951 1784186801.743168 MIRROR_RESYNC 4 mirrors 000952 1784186802.886044 write 0x00fccd thru 0x0174d3 (0x7807 bytes) 000955 1784186802.968162 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39915): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39915): Mirror 3: 0xfbd179 (stale) Mirror 4: 0xfbd179 (stale) Mirror 5: 0xfbd179 (stale) CRC-32 checksum value for chunk [0, 0x39915): Mirror 1: 0x400b6e3f Mirror 6: 0x400b6e3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39915: skipped 000956 1784186804.356309 mapread 0x02bf88 thru 0x035cea (0x9d63 bytes) 000957 1784186804.467146 mapread 0x007973 thru 0x011194 (0x9822 bytes) 000958 1784186804.538304 mapread 0x0080d1 thru 0x00ab7f (0x2aaf bytes) 000960 1784186804.599205 mapread 0x01b2ea thru 0x027420 (0xc137 bytes) 000962 1784186804.679635 read 0x02347e thru 0x02dd59 (0xa8dc bytes) 000965 1784186804.760054 mapread 0x032246 thru 0x0371bd (0x4f78 bytes) 000966 1784186804.804902 mapread 0x03177e thru 0x032b00 (0x1383 bytes) 000970 1784186804.913436 trunc from 0x039915 to 0x02b8a1 (0xe073 bytes) 000972 1784186805.103561 trunc from 0x02b8a2 to 0x03755a (0xbcb9 bytes) 000973 1784186805.179062 trunc from 0x03755b to 0x00c1dd (0x2b37d bytes) 000975 1784186805.204870 read 0x000116 thru 0x00c1dd (0xc0c8 bytes) 000977 1784186805.218019 read 0x0030b3 thru 0x00c1dd (0x912b bytes) 000979 1784186805.221007 mapread 0x00324d thru 0x00b90c (0x86c0 bytes) 000980 1784186805.229163 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc1de): Mirror 3: 0xba03e17b (stale) Mirror 5: 0xba03e17b (stale) Mirror 6: 0xba03e17b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc1de): Mirror 3: 0xba03e17b (stale) Mirror 5: 0xba03e17b (stale) Mirror 6: 0xba03e17b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc1de: skipped 000981 1784186805.585625 write 0x03f43f thru 0x03ffff (0xbc1 bytes) 000983 1784186805.591947 read 0x0372c2 thru 0x03ffff (0x8d3e bytes) 000984 1784186805.600387 mapread 0x03b9de thru 0x03ea78 (0x309b bytes) 000985 1784186805.606461 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb257906f (stale) Mirror 6: 0x4f2057eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb257906f (stale) Mirror 6: 0x4f2057eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1784186806.037893 write 0x00732f thru 0x00dee9 (0x6bbb bytes) 000991 1784186806.078200 read 0x02ef9c thru 0x0336e6 (0x474b bytes) 000993 1784186806.117818 read 0x025784 thru 0x033910 (0xe18d bytes) 000996 1784186806.152871 write 0x033f3f thru 0x037862 (0x3924 bytes) 000999 1784186806.158053 mapread 0x0369f6 thru 0x0397fa (0x2e05 bytes) All operations completed A-OK! PASS 70a (90s) == sanity-flr test 71: check mirror extend parent fid ==== 03:26:52 (1784186812) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 03:26:56 (1784186816) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.266915 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301168 s, 3.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x273:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x26f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x273:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x272:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x270:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 03:27:07 (1784186827) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7302432 .. Extending file size to 4692352 .. Extending file size to 4699168 .. Extending file size to 7936928 .. Extending file size to 5080352 .. Extending file size to 8033216 .. Extending file size to 6543584 .. Extending file size to 8060096 .. Extending file size to 8709248 .. Extending file size to 8736320 .. Extending file size to 6626144 .. Extending file size to 7009216 .. Extending file size to 8440352 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8703200 .. Extending file size to 8826816 .. Extending file size to 9069184 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 56877 56878 56879 56881 56883 done 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) 192.168.202.144@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:28:29 (1784186909) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7105856 .. Extending file size to 2551520 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6399680 .. failed Extending file size to 7442816 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8399520 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8669280 .. Extending file size to 9192768 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8701184 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9041760 .. Waiting 58489 58490 58491 58492 58493 failed 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) 192.168.202.144@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:29:53 (1784186993) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173644 s, 12.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.394801 s, 5.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1a88b7d0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) PASS 200c (25s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:30:20 (1784187020) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.184448 s, 11.4 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 03:30:29 (1784187029) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x197:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x289:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x285:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:30:37 (1784187037) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.353512 s, 5.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x286:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x287:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:30:46 (1784187046) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274028 s, 7.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x288:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x289:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:30:56 (1784187056) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244355 s, 8.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28b:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:31:33 (1784187093) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg244-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201158 s, 5.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:32:36 (1784187156) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg244-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13837 s, 15.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x28f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:33:40 (1784187220) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg244-server mds-ost sync done. Creating new pool test_204f oleg244-server: Pool lustre.test_204f created Adding targets to pool oleg244-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg244-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.308571 s, 6.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x290:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x290:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg244-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg244-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg244-server: Pool lustre.test_204f destroyed PASS 204f (86s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:35:05 (1784187305) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:35:13 (1784187313) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:35:21 (1784187321) oleg244-server: Pool lustre.test_206 created Adding targets to pool oleg244-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg244-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg244-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg244-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg244-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:35:49 (1784187349) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148978 s, 7.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783607 s, 13.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc2251bba Mirror 3: 0xc2251bba PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:36:03 (1784187363) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:36:07 (1784187367) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:36:12 (1784187372) Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.129129 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xbc0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x9d0000 Read this (4 bytes): BBBB 192.168.202.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg244-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:36:50 (1784187410) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0266629 s, 307 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:36:58 (1784187418) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489815 s, 21.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1bc lmm_fid: [0x200000404:0x1bc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1bc lmm_fid: [0x200000404:0x1bc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834364 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-16 03:37:00.000000000 -0400 Modify: 2026-07-16 03:37:00.000000000 -0400 Change: 2026-07-16 03:37:02.000000000 -0400 Birth: 2026-07-16 03:37:00.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:37:08 (1784187428) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:37:12 (1784187432) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.731251 s, 56.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0440408 s, 93.0 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 16 03:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1be:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 16 03:37 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:37:23 (1784187443) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:37:31 (1784187451) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1888 sec =========== 03:37:44 (1784187464) === sanity-flr: start cleanup 03:37:46 (1784187466) === === sanity-flr: finish cleanup 03:37:53 (1784187473) ===