-----============= acceptance-small: sanity-flr ============----- Sun Sep 6 21:55:04 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg226-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 21:55:24 (1788746124) === oleg226-client.virtnet: executing check_config_client /mnt/lustre oleg226-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg226-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff984c91961000.idle_timeout=debug osc.lustre-OST0001-osc-ffff984c91961000.idle_timeout=debug disable quota as required oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 21:55:56 (1788746156) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7627] == sanity-flr test 0a: lfs mirror create with -N option == 21:55:59 (1788746159) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:56:16 (1788746176) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:56:20 (1788746180) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:56:24 (1788746184) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:56:50 (1788746210) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:56:54 (1788746214) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 21:56:58 (1788746218) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff984c91961000.stats=clear osc.lustre-OST0001-osc-ffff984c91961000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:57:07 (1788746227) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:57:21 (1788746241) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:57:30 (1788746250) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:57:34 (1788746254) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:57:42 (1788746262) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 21:57:52 (1788746272) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:58:00 (1788746280) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:58:03 (1788746283) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:58:11 (1788746291) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:58:21 (1788746301) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:58:28 (1788746308) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326951 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13939 s, 7.5 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:58:38 (1788746318) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.474378 s, 37.6 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:58:48 (1788746328) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (14s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:59:02 (1788746342) Stopping /mnt/lustre-ost1 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff984c91961000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff984c91961000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg226-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff984c91961000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff984c91961000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg226-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:00:19 (1788746419) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.498981 s, 16.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff984c91961000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff984c91961000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg226-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:01:02 (1788746462) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff984c91961000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff984c91961000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg226-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff984c91961000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff984c91961000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg226-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:02:23 (1788746543) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.18229 s, 8.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 6 22:02 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.240s user 0m0.000s sys 0m0.166s Stopping /mnt/lustre-ost1 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff984c91961000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff984c91961000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.113s user 0m0.001s sys 0m0.165s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg226-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg226-server oleg226-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff984c91961000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff984c91961000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.231s user 0m0.000s sys 0m0.188s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg226-server: oleg226-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg226-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg226-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff984c91961000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff984c91961000.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:04:00 (1788746640) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:04:04 (1788746644) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:04:08 (1788746648) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:04:12 (1788746652) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 22:04:19 (1788746659) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.526244 s, 23.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:04:31 (1788746671) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144976 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159356 s, 65.8 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:04:38 (1788746678) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:04:46 (1788746686) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00117792 s, 869 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138007 s, 742 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108226 s, 946 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109613 s, 934 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 22:04:57 (1788746697) create test files with size 15737506 .. 6f396b7325d8a504f02ed648148d7b83 - a41d064f2817f54fef3ccbd0d693cc08 - 6e126c6025d7b09ac17e82cd257a900f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg226-client.virtnet /mnt/lustre (opts:) Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 22:05:17 (1788746717) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:05:32 (1788746732) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:05:38 (1788746738) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997241 s, 10.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420096 s, 25.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 22:05:46 (1788746746) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128172 s, 16.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.24279 s, 17.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.179036 s, 11.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 22:06:01 (1788746761) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:06:04 (1788746764) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:06:07 (1788746767) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365561 s, 28.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 22:06:14 (1788746774) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:06:18 (1788746778) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:06:28 (1788746788) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.300447 s, 34.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg226-client.virtnet /mnt/lustre (opts:) Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:06:40 (1788746800) llite.lustre-ffff984c82aac000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.439051 s, 23.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg226-client.virtnet /mnt/lustre (opts:) Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff984ca0644800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:06:55 (1788746815) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:07:32 (1788746852) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:08:06 (1788746886) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:08:09 (1788746889) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.233503 s, 13.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.207952 s, 15.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x323c31d5 (nosync,stale) Mirror 2: 0x323c31d5 (nosync,stale) ** verify mirror 2 doesn't change original checksum: fda8317085e7969ac0033c31842e5202 - mirror 1 checksum: 59b4c890c59e7b3c67832216f9400360 - mirror 2 checksum: fda8317085e7969ac0033c31842e5202 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T22:08:12-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T22:08:12-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: fda8317085e7969ac0033c31842e5202 - mirror 1 checksum: 59b4c890c59e7b3c67832216f9400360 - mirror 2 checksum: 59b4c890c59e7b3c67832216f9400360 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T22:08:12-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T22:08:12-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:08:22 (1788746902) Failing mds1 on oleg226-server Stopping /mnt/lustre-mds1 (opts:) on oleg226-server 22:08:28 (1788746908) shut down facet: mds1 facet_host: oleg226-server facet_failover_host: oleg226-server Failover mds1 to oleg226-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 22:08:59 (1788746939) targets are mounted 22:08:59 (1788746939) facet_failover done oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506132 s, 20.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:09:17 (1788746957) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:09:20 (1788746960) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:09:23 (1788746963) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00520236 s, 3.1 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:09:30 (1788746970) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:09:33 (1788746973) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00938937 s, 436 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0696082 s, 58.8 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:09:39 (1788746979) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:09:59 (1788746999) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:10:26 (1788747026) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:10:48 (1788747048) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:11:11 (1788747071) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 22:11:18 (1788747078) mirror create pid 54328 mirror split pid 54330 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54328 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54330 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 22:12:25 (1788747145) Seed set to 7147 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1788747147.395569 mapwrite 0x006cc1 thru 0x00fc3e (0x8f7e bytes) truncating to largest ever: 0x76a4 000003 1788747147.480622 trunc from 0x00fc3f to 0x0076a3 (0x859b bytes) truncating to largest ever: 0xf4d4 000005 1788747147.494220 trunc from 0x0076a4 to 0x00f4d3 (0x7e30 bytes) 000007 1788747147.517057 read 0x008f18 thru 0x00f4d3 (0x65bc bytes) 000009 1788747147.530727 write 0x01044f thru 0x015083 (0x4c35 bytes) 000012 1788747147.534162 mapread 0x00f1c1 thru 0x00fa11 (0x851 bytes) 000013 1788747147.535211 read 0x00dcb5 thru 0x015083 (0x73cf bytes) 000014 1788747147.536408 mapwrite 0x033699 thru 0x03471e (0x1086 bytes) 000015 1788747147.632241 read 0x016619 thru 0x01b406 (0x4dee bytes) 000016 1788747147.633032 mapwrite 0x01f9b5 thru 0x025642 (0x5c8e bytes) 000018 1788747147.679881 mapread 0x008a76 thru 0x00d6f8 (0x4c83 bytes) 000023 1788747147.685460 read 0x00d36c thru 0x016ca5 (0x993a bytes) 000025 1788747147.687087 mapread 0x030421 thru 0x03471e (0x42fe bytes) 000026 1788747147.688838 mapwrite 0x0028d7 thru 0x004064 (0x178e bytes) 000027 1788747147.726612 mapread 0x027bbb thru 0x033f2d (0xc373 bytes) 000028 1788747147.728117 mapwrite 0x01636a thru 0x01c609 (0x62a0 bytes) 000029 1788747147.794283 read 0x033e42 thru 0x03471e (0x8dd bytes) 000030 1788747147.795441 read 0x021e0c thru 0x0294dc (0x76d1 bytes) 000032 1788747147.798300 read 0x01cdd4 thru 0x02c9b1 (0xfbde bytes) 000033 1788747147.799106 punch from 0x01d73e to 0x0289ad (0xb270 bytes) 000034 1788747147.809161 mapread 0x00a606 thru 0x017392 (0xcd8d bytes) 000037 1788747147.811739 mapread 0x004632 thru 0x00b375 (0x6d44 bytes) 000041 1788747147.822838 read 0x0294b5 thru 0x02cffb (0x3b47 bytes) 000042 1788747147.824575 punch from 0x02d030 to 0x03471e (0x76ef bytes) 000043 1788747147.831099 mapwrite 0x00e461 thru 0x016656 (0x81f6 bytes) 000045 1788747147.875541 mapread 0x007582 thru 0x0157cc (0xe24b bytes) 000046 1788747147.877445 mapread 0x02c3ae thru 0x02de4a (0x1a9d bytes) 000048 1788747147.891113 write 0x03c3eb thru 0x03e755 (0x236b bytes) truncating to largest ever: 0x3c0a7 000049 1788747147.892919 trunc from 0x03e756 to 0x03c0a6 (0x26af bytes) 000051 1788747147.915379 mapwrite 0x03bb3b thru 0x03ffff (0x44c5 bytes) 000053 1788747148.001522 MIRROR_EXTEND to 2 mirrors 000057 1788747148.614844 mapread 0x03d136 thru 0x03ffff (0x2eca bytes) truncating to largest ever: 0x3f1bb 000059 1788747148.644199 trunc from 0x040000 to 0x03f1ba (0xe45 bytes) 000060 1788747148.689874 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) 000062 1788747148.804926 write 0x0210ba thru 0x02c570 (0xb4b7 bytes) 000064 1788747148.818342 write 0x03fbcd thru 0x03ffff (0x433 bytes) 000066 1788747148.822469 mapwrite 0x0122db thru 0x01f978 (0xd69e bytes) 000067 1788747148.925992 punch from 0x000603 to 0x000878 (0x276 bytes) 000071 1788747148.935278 mapwrite 0x036e8e thru 0x03eb5a (0x7ccd bytes) 000072 1788747149.014119 mapwrite 0x01b4f0 thru 0x020f20 (0x5a31 bytes) 000077 1788747149.058076 read 0x02817f thru 0x032f5c (0xadde bytes) 000078 1788747149.061302 trunc from 0x040000 to 0x026456 (0x19ba9 bytes) 000080 1788747149.074859 trunc from 0x026457 to 0x022547 (0x3f0f bytes) 000081 1788747149.092201 mapread 0x018606 thru 0x020703 (0x80fe bytes) 000083 1788747149.094418 read 0x00f054 thru 0x01a4d7 (0xb484 bytes) 000085 1788747149.096439 mapread 0x0220e4 thru 0x022547 (0x464 bytes) 000086 1788747149.097516 read 0x0163e7 thru 0x0177d4 (0x13ee bytes) 000087 1788747149.099047 read 0x00f1f9 thru 0x0185a7 (0x93af bytes) 000088 1788747149.106198 MIRROR_EXTEND to 2 mirrors 000090 1788747149.900325 write 0x03a2f1 thru 0x03c163 (0x1e73 bytes) 000091 1788747149.962452 mapread 0x007069 thru 0x00b79c (0x4734 bytes) 000095 1788747149.990342 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) 000096 1788747150.156361 mapwrite 0x02d3a6 thru 0x036c06 (0x9861 bytes) 000098 1788747150.226278 write 0x021c66 thru 0x02298e (0xd29 bytes) 000101 1788747150.236950 write 0x0034e7 thru 0x008783 (0x529d bytes) 000103 1788747150.246661 mapwrite 0x017333 thru 0x01c9a3 (0x5671 bytes) 000104 1788747150.297501 punch from 0x0379ed to 0x03c163 (0x4777 bytes) 000105 1788747150.303526 read 0x010dfa thru 0x01656d (0x5774 bytes) 000107 1788747150.308422 read 0x001d11 thru 0x007f23 (0x6213 bytes) 000108 1788747150.325078 mapwrite 0x0181b8 thru 0x01b152 (0x2f9b bytes) 000110 1788747150.382296 trunc from 0x03c164 to 0x00585f (0x36904 bytes) 000111 1788747150.444427 trunc from 0x005860 to 0x02829b (0x22a3c bytes) 000112 1788747150.466029 trunc from 0x02829c to 0x010f36 (0x17365 bytes) 000113 1788747150.499390 trunc from 0x010f37 to 0x031d61 (0x20e2b bytes) 000115 1788747150.538263 punch from 0x016165 to 0x01a88e (0x472a bytes) 000116 1788747150.543407 read 0x017b42 thru 0x01beb1 (0x4370 bytes) 000117 1788747150.617426 read 0x02f690 thru 0x0304a7 (0xe18 bytes) 000119 1788747150.618368 read 0x001b11 thru 0x00a672 (0x8b62 bytes) 000120 1788747150.618921 mapread 0x01b411 thru 0x01bce2 (0x8d2 bytes) 000121 1788747150.619673 mapwrite 0x0084fe thru 0x01196c (0x946f bytes) 000123 1788747150.670577 read 0x026dd5 thru 0x031d61 (0xaf8d bytes) 000124 1788747150.671241 punch from 0x00ce50 to 0x019b8c (0xcd3d bytes) 000125 1788747150.676005 trunc from 0x031d62 to 0x03aa69 (0x8d08 bytes) 000126 1788747150.695360 read 0x01dc90 thru 0x02aaac (0xce1d bytes) 000127 1788747150.696337 mapwrite 0x00c757 thru 0x01b0a1 (0xe94b bytes) 000128 1788747150.767304 mapread 0x01b0d7 thru 0x01f676 (0x45a0 bytes) 000130 1788747150.771307 mapwrite 0x01efe7 thru 0x0237d0 (0x47ea bytes) 000131 1788747150.838562 punch from 0x005dee to 0x00adf4 (0x5007 bytes) 000132 1788747150.851630 mapwrite 0x010dc4 thru 0x01da5d (0xcc9a bytes) 000133 1788747150.911944 mapread 0x00a6aa thru 0x015547 (0xae9e bytes) 000134 1788747150.917886 read 0x0314ed thru 0x0393c3 (0x7ed7 bytes) 000137 1788747150.922455 mapread 0x013f24 thru 0x01f1db (0xb2b8 bytes) 000138 1788747150.923705 mapwrite 0x00c156 thru 0x0120d1 (0x5f7c bytes) 000139 1788747150.991520 mapwrite 0x017be9 thru 0x01dd9b (0x61b3 bytes) 000141 1788747151.047283 read 0x00e453 thru 0x01b2f5 (0xcea3 bytes) 000147 1788747151.053336 write 0x014424 thru 0x021149 (0xcd26 bytes) 000148 1788747151.055254 punch from 0x00baad to 0x01af6a (0xf4be bytes) 000149 1788747151.073406 trunc from 0x03aa6a to 0x004876 (0x361f3 bytes) 000155 1788747151.096374 write 0x01d0a2 thru 0x01dc04 (0xb63 bytes) 000157 1788747151.098439 mapwrite 0x002f99 thru 0x0114c2 (0xe52a bytes) 000160 1788747151.157710 mapwrite 0x027bc5 thru 0x0316c3 (0x9aff bytes) 000161 1788747151.222385 mapread 0x007625 thru 0x00af80 (0x395c bytes) 000162 1788747151.224912 trunc from 0x0316c4 to 0x021dd4 (0xf8ef bytes) 000163 1788747151.243999 punch from 0x00957a to 0x0140d3 (0xab5a bytes) 000169 1788747151.262908 MIRROR_EXTEND to 2 mirrors 000173 1788747151.968890 trunc from 0x021dd5 to 0x002012 (0x1fdc2 bytes) 000176 1788747152.120204 punch from 0x0005ae to 0x002012 (0x1a65 bytes) 000177 1788747152.125457 mapwrite 0x01deb2 thru 0x02dac8 (0xfc17 bytes) 000178 1788747152.243552 read 0x00dbcf thru 0x00e099 (0x4cb bytes) 000179 1788747152.255582 mapwrite 0x03c4ec thru 0x03e562 (0x2077 bytes) 000180 1788747152.353385 write 0x02d939 thru 0x03148b (0x3b53 bytes) 000182 1788747152.360677 write 0x0375fc thru 0x03d001 (0x5a06 bytes) 000183 1788747152.362891 mapwrite 0x013dc5 thru 0x0156b3 (0x18ef bytes) 000184 1788747152.414518 trunc from 0x03e563 to 0x0291a9 (0x153b9 bytes) 000185 1788747152.437516 trunc from 0x0291aa to 0x017fa3 (0x11206 bytes) 000186 1788747152.449425 write 0x015d30 thru 0x02262a (0xc8fb bytes) 000187 1788747152.451298 read 0x02075a thru 0x02262a (0x1ed1 bytes) 000188 1788747152.452978 trunc from 0x02262b to 0x00bc14 (0x16a16 bytes) 000189 1788747152.478243 MIRROR_RESYNC 2 mirrors 000191 1788747153.090897 write 0x034b73 thru 0x03f64b (0xaad9 bytes) 000192 1788747153.174232 punch from 0x026d04 to 0x02cd7c (0x6079 bytes) 000193 1788747153.203235 punch from 0x03a742 to 0x03f64b (0x4f0a bytes) 000194 1788747153.208268 punch from 0x02c184 to 0x03aae1 (0xe95e bytes) 000195 1788747153.213647 trunc from 0x03f64c to 0x0036b6 (0x3bf95 bytes) 000196 1788747153.255887 read 0x00132b thru 0x0036b6 (0x238c bytes) 000197 1788747153.261495 trunc from 0x0036b7 to 0x0140c5 (0x10a0f bytes) 000198 1788747153.286249 mapwrite 0x00b1a3 thru 0x016161 (0xafbf bytes) 000199 1788747153.388232 punch from 0x00f11d to 0x016161 (0x7045 bytes) 000200 1788747153.393247 write 0x00f173 thru 0x01e13b (0xefc9 bytes) 000201 1788747153.403620 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) 000203 1788747153.560290 read 0x01791a thru 0x01e13b (0x6822 bytes) 000204 1788747153.575245 trunc from 0x01e13c to 0x006aaf (0x1768c bytes) 000206 1788747153.598449 mapread 0x002c06 thru 0x00500a (0x2405 bytes) 000207 1788747153.615842 mapread 0x0026c3 thru 0x006aaf (0x43ed bytes) 000208 1788747153.617333 punch from 0x0050a5 to 0x006aaf (0x1a0b bytes) 000209 1788747153.623577 read 0x00271f thru 0x003ccc (0x15ae bytes) 000210 1788747153.625540 read 0x000056 thru 0x001548 (0x14f3 bytes) 000211 1788747153.636059 mapread 0x000e5e thru 0x006398 (0x553b bytes) 000212 1788747153.636978 mapwrite 0x01b777 thru 0x01f4da (0x3d64 bytes) 000213 1788747153.705858 trunc from 0x01f4db to 0x001a90 (0x1da4a bytes) 000215 1788747153.729468 punch from 0x000849 to 0x001a90 (0x1248 bytes) 000216 1788747153.733958 mapread 0x001a79 thru 0x001a90 (0x18 bytes) 000217 1788747153.738156 trunc from 0x001a91 to 0x000c2f (0xe61 bytes) 000218 1788747153.757196 read 0x000ac1 thru 0x000c2f (0x16f bytes) 000220 1788747153.768087 mapread 0x000006 thru 0x000c2f (0xc2a bytes) 000222 1788747153.769181 punch from 0x000308 to 0x000c2f (0x928 bytes) 000224 1788747153.785253 mapwrite 0x0187c9 thru 0x01d0fa (0x4932 bytes) 000225 1788747153.875718 read 0x0160bd thru 0x01d0fa (0x703e bytes) 000230 1788747153.880185 write 0x0256e4 thru 0x029018 (0x3935 bytes) 000233 1788747153.883704 write 0x01d8a8 thru 0x02bb4f (0xe2a8 bytes) 000234 1788747153.886191 read 0x01cf40 thru 0x029a5c (0xcb1d bytes) 000236 1788747153.888003 write 0x022135 thru 0x02f075 (0xcf41 bytes) 000238 1788747153.891267 mapread 0x02e8de thru 0x02f075 (0x798 bytes) 000239 1788747153.892711 write 0x002ac1 thru 0x00babd (0x8ffd bytes) 000241 1788747153.899443 read 0x025748 thru 0x02f075 (0x992e bytes) 000243 1788747153.900982 mapread 0x027614 thru 0x02f075 (0x7a62 bytes) 000244 1788747153.902400 trunc from 0x02f076 to 0x017631 (0x17a44 bytes) 000245 1788747153.917781 mapwrite 0x024da4 thru 0x032cb8 (0xdf15 bytes) 000246 1788747154.007661 write 0x00339f thru 0x00e7af (0xb411 bytes) 000247 1788747154.009003 trunc from 0x032cb9 to 0x01672d (0x1c58b bytes) 000248 1788747154.025265 punch from 0x001380 to 0x00eb3e (0xd7bf bytes) 000249 1788747154.044932 mapread 0x007087 thru 0x00d828 (0x67a2 bytes) 000250 1788747154.073435 punch from 0x0164b5 to 0x01672d (0x279 bytes) 000251 1788747154.078220 mapread 0x008ec9 thru 0x00bf8b (0x30c3 bytes) 000255 1788747154.080065 mapread 0x00e319 thru 0x01672d (0x8415 bytes) 000256 1788747154.086951 trunc from 0x01672e to 0x026925 (0x101f8 bytes) 000257 1788747154.103314 punch from 0x016375 to 0x01c54e (0x61da bytes) 000258 1788747154.111220 mapread 0x0051de thru 0x00836c (0x318f bytes) 000260 1788747154.123791 mapwrite 0x00bdff thru 0x0197f2 (0xd9f4 bytes) 000262 1788747154.186336 read 0x00d95c thru 0x00f1de (0x1883 bytes) 000264 1788747154.188842 punch from 0x01ce60 to 0x01d39f (0x540 bytes) 000269 1788747154.196520 read 0x00ba17 thru 0x014b6c (0x9156 bytes) 000270 1788747154.197433 trunc from 0x026926 to 0x018979 (0xdfac bytes) 000272 1788747154.213116 read 0x00a924 thru 0x017ecf (0xd5ac bytes) 000273 1788747154.213973 mapwrite 0x00ab68 thru 0x012f85 (0x841e bytes) 000274 1788747154.258462 mapread 0x013393 thru 0x018979 (0x55e7 bytes) 000280 1788747154.262934 write 0x00a52c thru 0x016a83 (0xc558 bytes) 000281 1788747154.264892 write 0x036b0c thru 0x03c66d (0x5b62 bytes) 000283 1788747154.266850 mapwrite 0x02dc08 thru 0x02ed84 (0x117d bytes) 000284 1788747154.309852 trunc from 0x03c66e to 0x021605 (0x1b068 bytes) 000285 1788747154.333099 MIRROR_EXTEND to 2 mirrors 000287 1788747155.024126 write 0x02e078 thru 0x0373ba (0x9343 bytes) 000288 1788747155.074633 trunc from 0x0373bb to 0x015396 (0x22024 bytes) 000289 1788747155.099167 punch from 0x002b11 to 0x009f55 (0x7445 bytes) 000290 1788747155.108186 trunc from 0x015397 to 0x0296e9 (0x14353 bytes) 000291 1788747155.120482 mapread 0x00dee7 thru 0x0169c0 (0x8ada bytes) 000294 1788747155.139430 write 0x00831c thru 0x008546 (0x22b bytes) 000295 1788747155.145651 MIRROR_RESYNC 2 mirrors 000296 1788747155.698125 read 0x01f4ae thru 0x020b56 (0x16a9 bytes) 000297 1788747155.701620 trunc from 0x0296ea to 0x03b8e2 (0x121f9 bytes) 000301 1788747155.795823 read 0x02a2fa thru 0x037b86 (0xd88d bytes) 000302 1788747155.805902 trunc from 0x03b8e3 to 0x02e112 (0xd7d0 bytes) 000303 1788747155.825074 read 0x01f2f5 thru 0x0239c3 (0x46cf bytes) 000304 1788747155.826286 read 0x028552 thru 0x02e112 (0x5bc1 bytes) 000305 1788747155.827272 mapread 0x00f898 thru 0x01cc49 (0xd3b2 bytes) 000306 1788747155.829137 write 0x03cabc thru 0x03ffff (0x3544 bytes) 000307 1788747155.830718 trunc from 0x040000 to 0x028dc3 (0x1723c bytes) 000310 1788747155.845025 punch from 0x01c4e4 to 0x01d09e (0xbbb bytes) 000311 1788747155.849014 punch from 0x00effa to 0x01603c (0x7043 bytes) 000312 1788747155.854939 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) 000314 1788747155.980954 trunc from 0x028dc4 to 0x02f341 (0x657e bytes) 000315 1788747155.998542 trunc from 0x02f342 to 0x02909e (0x62a3 bytes) 000317 1788747156.019016 write 0x016b69 thru 0x01f550 (0x89e8 bytes) 000319 1788747156.032358 MIRROR_EXTEND to 2 mirrors 000320 1788747156.884023 mapread 0x01b8d4 thru 0x01ee93 (0x35c0 bytes) 000321 1788747156.916692 MIRROR_EXTEND to 3 mirrors 000323 1788747157.675130 trunc from 0x02909f to 0x02a09b (0xffd bytes) 000324 1788747157.753946 write 0x01407c thru 0x020ee3 (0xce68 bytes) 000326 1788747157.765333 write 0x02319d thru 0x030670 (0xd4d4 bytes) 000327 1788747157.782055 mapread 0x020db6 thru 0x024663 (0x38ae bytes) 000331 1788747157.806237 trunc from 0x030671 to 0x0080b3 (0x285bd bytes) 000334 1788747157.833801 mapread 0x00126d thru 0x005983 (0x4717 bytes) 000335 1788747157.848667 mapread 0x004387 thru 0x0080b3 (0x3d2d bytes) 000336 1788747157.849585 mapread 0x007998 thru 0x0080b3 (0x71c bytes) 000338 1788747157.850636 write 0x008a39 thru 0x0118de (0x8ea6 bytes) 000339 1788747157.852585 mapwrite 0x03a82c thru 0x03ffff (0x57d4 bytes) 000340 1788747157.951705 read 0x03ac9b thru 0x03ffff (0x5365 bytes) 000341 1788747157.952432 trunc from 0x040000 to 0x032f8c (0xd073 bytes) 000342 1788747157.971566 trunc from 0x032f8d to 0x0170f6 (0x1be96 bytes) 000343 1788747158.014999 punch from 0x003d07 to 0x005990 (0x1c8a bytes) 000344 1788747158.022427 write 0x03689b thru 0x036aba (0x220 bytes) 000345 1788747158.035276 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36abb): Mirror 3: 0x970497a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36abb): Mirror 3: 0x970497a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36abb: skipped 000346 1788747158.409348 trunc from 0x036abb to 0x005798 (0x31322 bytes) 000348 1788747158.462625 MIRROR_RESYNC 2 mirrors 000349 1788747159.176275 mapread 0x001343 thru 0x005798 (0x4456 bytes) 000351 1788747159.190840 read 0x00426c thru 0x005798 (0x152d bytes) 000352 1788747159.191289 read 0x003966 thru 0x00409e (0x739 bytes) 000353 1788747159.195724 MIRROR_EXTEND to 3 mirrors 000355 1788747159.744404 trunc from 0x005799 to 0x0217c0 (0x1c028 bytes) 000359 1788747159.837441 write 0x03f9a8 thru 0x03ffff (0x658 bytes) 000360 1788747159.839259 read 0x03183b thru 0x037f4b (0x6711 bytes) 000361 1788747159.851974 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7c370f42 Mirror 5: 0x7c370f42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000363 1788747160.866218 mapread 0x00a5d8 thru 0x016ba8 (0xc5d1 bytes) 000365 1788747160.896136 mapread 0x03ddca thru 0x03ffff (0x2236 bytes) 000369 1788747160.925057 trunc from 0x040000 to 0x037e8e (0x8171 bytes) 000370 1788747160.978295 punch from 0x026546 to 0x0325c2 (0xc07d bytes) 000371 1788747160.981362 mapwrite 0x01bbfe thru 0x021247 (0x564a bytes) 000373 1788747161.024895 read 0x0102ee thru 0x014e75 (0x4b88 bytes) 000374 1788747161.029672 mapwrite 0x0285a6 thru 0x033667 (0xb0c2 bytes) 000376 1788747161.065600 mapwrite 0x023995 thru 0x030071 (0xc6dd bytes) 000377 1788747161.094437 punch from 0x029741 to 0x037e8e (0xe74e bytes) 000378 1788747161.097953 mapread 0x0326ed thru 0x036850 (0x4164 bytes) 000379 1788747161.103723 read 0x0254fb thru 0x030f36 (0xba3c bytes) 000380 1788747161.112803 MIRROR_RESYNC 4 mirrors 000381 1788747161.931611 trunc from 0x037e8f to 0x00fe49 (0x28045 bytes) 000382 1788747162.096574 trunc from 0x00fe4a to 0x03a90f (0x2aac6 bytes) 000384 1788747162.124879 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a910): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a910): Mirror 3: 0x89199441 (stale) Mirror 4: 0x89199441 (stale) Mirror 5: 0x89199441 (stale) CRC-32 checksum value for chunk [0, 0x3a910): Mirror 1: 0xb0a0c22e Mirror 6: 0xb0a0c22e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a910: skipped 000386 1788747163.111389 punch from 0x02c130 to 0x02ced9 (0xdaa bytes) 000388 1788747163.197228 punch from 0x01fb10 to 0x020b31 (0x1022 bytes) 000389 1788747163.203697 trunc from 0x03a910 to 0x02afcf (0xf940 bytes) 000390 1788747163.229967 mapread 0x0115a3 thru 0x0116ca (0x128 bytes) 000393 1788747163.236112 read 0x008472 thru 0x00dd5b (0x58ea bytes) 000395 1788747163.247380 trunc from 0x02afd0 to 0x0134a9 (0x17b26 bytes) 000396 1788747163.272080 read 0x0024b2 thru 0x00713d (0x4c8c bytes) 000398 1788747163.285437 trunc from 0x0134aa to 0x0276d0 (0x14227 bytes) 000399 1788747163.306254 write 0x00ab90 thru 0x01395b (0x8dcc bytes) 000400 1788747163.308171 write 0x00874d thru 0x00c691 (0x3f45 bytes) 000402 1788747163.312160 trunc from 0x0276d1 to 0x03b070 (0x139a0 bytes) 000405 1788747163.333093 mapwrite 0x010ca6 thru 0x0157d0 (0x4b2b bytes) 000406 1788747163.380276 punch from 0x02bf0d to 0x0334d9 (0x75cd bytes) 000408 1788747163.384729 write 0x014fa9 thru 0x01f532 (0xa58a bytes) 000409 1788747163.386047 mapwrite 0x006778 thru 0x015cfd (0xf586 bytes) 000410 1788747163.430734 read 0x01710f thru 0x020e73 (0x9d65 bytes) 000412 1788747163.441383 MIRROR_RESYNC 5 mirrors 000414 1788747164.618440 mapwrite 0x028def thru 0x03574e (0xc960 bytes) 000415 1788747164.929877 punch from 0x008d04 to 0x010a66 (0x7d63 bytes) 000416 1788747164.947385 mapread 0x00cbb0 thru 0x01117f (0x45d0 bytes) 000417 1788747164.979338 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b071): Mirror 3: 0x81ef961b (stale) Mirror 4: 0x81ef961b (stale) Mirror 5: 0x81ef961b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b071): Mirror 3: 0x81ef961b (stale) Mirror 4: 0x81ef961b (stale) Mirror 5: 0x81ef961b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b071: skipped 000418 1788747165.422476 trunc from 0x03b071 to 0x00dc99 (0x2d3d7 bytes) 000419 1788747165.448104 trunc from 0x00dc9a to 0x01f266 (0x115cd bytes) 000420 1788747165.464357 mapread 0x01d990 thru 0x01f266 (0x18d7 bytes) 000421 1788747165.490579 trunc from 0x01f267 to 0x010fea (0xe27c bytes) 000422 1788747165.511222 mapwrite 0x0050cb thru 0x009c05 (0x4b3b bytes) 000426 1788747165.572053 mapread 0x00f5b7 thru 0x010fea (0x1a34 bytes) 000427 1788747165.573387 punch from 0x0088b2 to 0x010fea (0x8739 bytes) 000428 1788747165.578512 mapread 0x0097c1 thru 0x010fea (0x782a bytes) 000429 1788747165.587613 mapread 0x00e4ec thru 0x010fea (0x2aff bytes) 000430 1788747165.589034 write 0x008930 thru 0x00afba (0x268b bytes) 000434 1788747165.593861 mapread 0x0055e6 thru 0x00f55f (0x9f7a bytes) 000436 1788747165.596166 punch from 0x00f0fb to 0x010fea (0x1ef0 bytes) 000440 1788747165.607394 write 0x00e5a1 thru 0x012094 (0x3af4 bytes) 000441 1788747165.608887 mapwrite 0x000220 thru 0x003625 (0x3406 bytes) 000443 1788747165.647114 read 0x00c136 thru 0x012094 (0x5f5f bytes) 000444 1788747165.647554 mapread 0x007a7e thru 0x00af3d (0x34c0 bytes) 000445 1788747165.651169 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x12095): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12095): Mirror 3: 0x901a8528 (stale) Mirror 4: 0x901a8528 (stale) Mirror 5: 0x901a8528 (stale) CRC-32 checksum value for chunk [0, 0x12095): Mirror 1: 0x6dd9fd9c Mirror 6: 0x6dd9fd9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12095: skipped 000448 1788747166.430393 mapread 0x000de6 thru 0x00f36e (0xe589 bytes) 000450 1788747166.475172 read 0x004738 thru 0x00dc2e (0x94f7 bytes) 000451 1788747166.486191 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x12095): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12095): Mirror 3: 0x901a8528 (stale) Mirror 4: 0x901a8528 (stale) Mirror 5: 0x901a8528 (stale) CRC-32 checksum value for chunk [0, 0x12095): Mirror 1: 0x6dd9fd9c Mirror 6: 0x6dd9fd9c Mirror 7: 0x6dd9fd9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12095: skipped 000452 1788747167.498245 mapread 0x009cc5 thru 0x012094 (0x83d0 bytes) 000453 1788747167.574198 mapwrite 0x00ebc6 thru 0x01ba4b (0xce86 bytes) 000454 1788747167.829968 mapwrite 0x002c09 thru 0x003b3d (0xf35 bytes) 000457 1788747167.897401 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1ba4c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ba4c): Mirror 3: 0x793280bf (stale) Mirror 4: 0x793280bf (stale) Mirror 5: 0x793280bf (stale) Mirror 6: 0x35fb1974 (stale) Mirror 7: 0x35fb1974 (stale) CRC-32 checksum value for chunk [0, 0x1ba4c): Mirror 1: 0xbae51763 Mirror 8: 0xbae51763 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ba4c: skipped 000458 1788747168.637499 punch from 0x00a60c to 0x00a9bc (0x3b1 bytes) 000459 1788747168.714113 trunc from 0x01ba4c to 0x037f64 (0x1c519 bytes) 000461 1788747168.732084 trunc from 0x037f65 to 0x025c28 (0x1233c bytes) 000462 1788747168.754235 punch from 0x015548 to 0x01cfbe (0x7a77 bytes) 000463 1788747168.758372 mapwrite 0x02df8f thru 0x039b43 (0xbbb5 bytes) 000464 1788747168.851529 write 0x02a49c thru 0x03795a (0xd4bf bytes) 000465 1788747168.852837 write 0x012874 thru 0x014d14 (0x24a1 bytes) 000466 1788747168.861583 read 0x030a56 thru 0x03472a (0x3cd5 bytes) 000469 1788747168.863943 trunc from 0x039b44 to 0x01a6ca (0x1f479 bytes) 000470 1788747168.882668 read 0x019250 thru 0x019942 (0x6f3 bytes) 000471 1788747168.896599 mapwrite 0x0300e3 thru 0x037d18 (0x7c36 bytes) 000472 1788747168.963692 trunc from 0x037d19 to 0x02bcd1 (0xc047 bytes) 000473 1788747168.977969 write 0x030616 thru 0x03c507 (0xbef2 bytes) 000474 1788747168.981790 mapwrite 0x004e15 thru 0x009673 (0x485f bytes) 000475 1788747169.019414 mapread 0x0299a0 thru 0x02c716 (0x2d77 bytes) 000476 1788747169.023317 mapwrite 0x033286 thru 0x03ffff (0xcd7a bytes) 000477 1788747169.083497 write 0x037820 thru 0x0388d3 (0x10b4 bytes) 000478 1788747169.084442 punch from 0x01ccaf to 0x022cd2 (0x6024 bytes) 000479 1788747169.091901 punch from 0x003144 to 0x00a45d (0x731a bytes) 000481 1788747169.096137 mapwrite 0x016dc6 thru 0x0245bf (0xd7fa bytes) 000482 1788747169.172928 MIRROR_RESYNC 7 mirrors 000483 1788747170.407335 mapwrite 0x00c01e thru 0x015914 (0x98f7 bytes) 000485 1788747170.770945 punch from 0x03b7c2 to 0x03ffff (0x483e bytes) 000489 1788747170.801492 MIRROR_RESYNC 7 mirrors 000492 1788747172.159259 write 0x0072f7 thru 0x00970f (0x2419 bytes) 000493 1788747172.250227 mapread 0x00a007 thru 0x00ffd0 (0x5fca bytes) 000494 1788747172.275485 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb281706f (stale) Mirror 4: 0xb281706f (stale) Mirror 5: 0xb281706f (stale) Mirror 6: 0xb281706f (stale) Mirror 7: 0xb281706f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb281706f (stale) Mirror 4: 0xb281706f (stale) Mirror 5: 0xb281706f (stale) Mirror 6: 0xb281706f (stale) Mirror 7: 0xb281706f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000495 1788747172.716038 trunc from 0x040000 to 0x00d953 (0x326ac bytes) 000497 1788747172.730333 mapread 0x007814 thru 0x00d953 (0x6140 bytes) 000499 1788747172.750562 write 0x037498 thru 0x039ef4 (0x2a5d bytes) 000500 1788747172.751820 mapread 0x035606 thru 0x039ef4 (0x48ef bytes) 000503 1788747172.759270 mapread 0x019312 thru 0x01fea1 (0x6b90 bytes) 000505 1788747172.766836 trunc from 0x039ef5 to 0x005ea2 (0x34052 bytes) 000506 1788747172.786993 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x5ea3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5ea3): Mirror 3: 0xbe0cb8be (stale) Mirror 4: 0xbe0cb8be (stale) Mirror 5: 0xbe0cb8be (stale) Mirror 6: 0xbe0cb8be (stale) Mirror 7: 0xbe0cb8be (stale) CRC-32 checksum value for chunk [0, 0x5ea3): Mirror 1: 0xbe0cb8be Mirror 8: 0xbe0cb8be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5ea3: skipped 000507 1788747173.628346 mapread 0x0025c0 thru 0x005ea2 (0x38e3 bytes) 000508 1788747173.670372 mapread 0x00365d thru 0x005ea2 (0x2846 bytes) 000510 1788747173.707677 mapwrite 0x0067a4 thru 0x00fbbb (0x9418 bytes) 000511 1788747173.914731 mapread 0x003c99 thru 0x006661 (0x29c9 bytes) 000512 1788747173.916425 punch from 0x0004de to 0x00fbbb (0xf6de bytes) 000513 1788747173.923253 read 0x009144 thru 0x00f7a7 (0x6664 bytes) 000514 1788747173.938661 write 0x029aef thru 0x029b3d (0x4f bytes) 000516 1788747173.940622 trunc from 0x029b3e to 0x01391c (0x16221 bytes) 000517 1788747173.969299 mapwrite 0x0298f1 thru 0x0332fb (0x9a0b bytes) 000519 1788747174.082995 trunc from 0x0332fc to 0x0093d7 (0x29f24 bytes) 000520 1788747174.115844 mapread 0x00885d thru 0x0093d7 (0xb7b bytes) 000521 1788747174.133126 trunc from 0x0093d8 to 0x0279cb (0x1e5f4 bytes) 000522 1788747174.155107 punch from 0x005d56 to 0x007fb6 (0x2261 bytes) 000524 1788747174.160300 mapwrite 0x02112f thru 0x02aea7 (0x9d79 bytes) 000525 1788747174.312902 write 0x00e3a8 thru 0x017c5c (0x98b5 bytes) 000526 1788747174.327420 mapread 0x0194a9 thru 0x01edc4 (0x591c bytes) 000527 1788747174.328475 read 0x02169c thru 0x02320d (0x1b72 bytes) 000528 1788747174.329897 punch from 0x020bef to 0x02aea7 (0xa2b9 bytes) 000530 1788747174.362832 MIRROR_RESYNC 7 mirrors 000531 1788747175.578280 trunc from 0x02aea8 to 0x02fb37 (0x4c90 bytes) 000532 1788747175.666134 mapwrite 0x00cd20 thru 0x00ee01 (0x20e2 bytes) 000534 1788747175.799270 mapwrite 0x00d600 thru 0x015bc3 (0x85c4 bytes) 000535 1788747175.842319 MIRROR_RESYNC 7 mirrors 000536 1788747176.806231 write 0x03b20a thru 0x03ffff (0x4df6 bytes) 000537 1788747176.918322 write 0x03fcc2 thru 0x03ffff (0x33e bytes) 000538 1788747176.919401 mapread 0x017d9e thru 0x0272b0 (0xf513 bytes) 000539 1788747176.952193 punch from 0x02c85e to 0x0384b5 (0xbc58 bytes) 000540 1788747176.963378 trunc from 0x040000 to 0x00e74b (0x318b4 bytes) 000541 1788747176.990081 read 0x002913 thru 0x00e74b (0xbe39 bytes) 000543 1788747176.997578 mapwrite 0x02feb0 thru 0x033079 (0x31ca bytes) 000544 1788747177.067108 mapread 0x01a3ea thru 0x01ea16 (0x462d bytes) 000545 1788747177.077770 mapwrite 0x024996 thru 0x033592 (0xebfd bytes) 000546 1788747177.154477 trunc from 0x033593 to 0x011cb2 (0x218e0 bytes) 000547 1788747177.173453 write 0x00c1fc thru 0x016f5b (0xad60 bytes) 000549 1788747177.176104 punch from 0x00947e to 0x0138a2 (0xa425 bytes) 000550 1788747177.197250 read 0x0146f3 thru 0x016f5b (0x2869 bytes) 000551 1788747177.197953 punch from 0x007746 to 0x016e2b (0xf6e6 bytes) 000552 1788747177.205681 trunc from 0x016f5c to 0x03f150 (0x281f5 bytes) 000553 1788747177.225524 mapwrite 0x03e8d1 thru 0x03ffff (0x172f bytes) 000554 1788747177.296601 read 0x011975 thru 0x01b6a8 (0x9d34 bytes) 000556 1788747177.298240 mapwrite 0x033c7d thru 0x035e5f (0x21e3 bytes) 000557 1788747177.342207 mapwrite 0x02bc27 thru 0x033ce8 (0x80c2 bytes) 000558 1788747177.393272 trunc from 0x040000 to 0x031cb1 (0xe34e bytes) 000559 1788747177.414256 MIRROR_RESYNC 7 mirrors 000560 1788747178.618900 write 0x03571f thru 0x03ffff (0xa8e1 bytes) 000561 1788747178.678165 write 0x02cd22 thru 0x0399f7 (0xccd6 bytes) 000562 1788747178.684517 punch from 0x00662a to 0x006732 (0x109 bytes) 000563 1788747178.698168 mapread 0x036624 thru 0x03bd17 (0x56f4 bytes) 000564 1788747178.699501 punch from 0x01cdb1 to 0x01e14b (0x139b bytes) 000565 1788747178.708251 MIRROR_RESYNC 7 mirrors 000566 1788747179.742208 MIRROR_RESYNC 7 mirrors 000567 1788747180.123304 mapwrite 0x00cdc8 thru 0x0118e4 (0x4b1d bytes) 000568 1788747180.255692 mapwrite 0x008238 thru 0x009928 (0x16f1 bytes) 000569 1788747180.293293 write 0x00ccea thru 0x012ba6 (0x5ebd bytes) 000570 1788747180.294351 write 0x03a927 thru 0x03bd30 (0x140a bytes) 000571 1788747180.303518 trunc from 0x040000 to 0x032415 (0xdbea bytes) 000572 1788747180.330034 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32416): Mirror 4: 0x50aae55e (stale) Mirror 5: 0x50aae55e (stale) Mirror 6: 0x50aae55e (stale) Mirror 7: 0x50aae55e (stale) Mirror 8: 0x50aae55e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32416): Mirror 4: 0x50aae55e (stale) Mirror 5: 0x50aae55e (stale) Mirror 6: 0x50aae55e (stale) Mirror 7: 0x50aae55e (stale) Mirror 8: 0x50aae55e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32416: skipped 000573 1788747180.936202 punch from 0x00b6c9 to 0x013b91 (0x84c9 bytes) 000575 1788747180.943383 punch from 0x0261d5 to 0x032415 (0xc241 bytes) 000576 1788747180.955816 punch from 0x000bd8 to 0x00baa6 (0xaecf bytes) 000579 1788747180.976719 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32416): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32416): Mirror 4: 0x50aae55e (stale) Mirror 5: 0x50aae55e (stale) Mirror 6: 0x50aae55e (stale) Mirror 7: 0x50aae55e (stale) Mirror 8: 0x50aae55e (stale) CRC-32 checksum value for chunk [0, 0x32416): Mirror 1: 0xc9a8a721 Mirror 9: 0xc9a8a721 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32416: skipped 000580 1788747182.126818 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32416): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32416): Mirror 4: 0x50aae55e (stale) Mirror 5: 0x50aae55e (stale) Mirror 6: 0x50aae55e (stale) Mirror 7: 0x50aae55e (stale) Mirror 8: 0x50aae55e (stale) CRC-32 checksum value for chunk [0, 0x32416): Mirror 1: 0xc9a8a721 Mirror 9: 0xc9a8a721 Mirror 10: 0xc9a8a721 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32416: skipped 000581 1788747182.971403 read 0x00fb26 thru 0x01799e (0x7e79 bytes) 000582 1788747182.983663 write 0x030803 thru 0x03a8cf (0xa0cd bytes) 000583 1788747183.066667 mapwrite 0x0154c0 thru 0x024912 (0xf453 bytes) 000585 1788747183.178098 write 0x01b263 thru 0x020f91 (0x5d2f bytes) 000586 1788747183.180682 mapread 0x023adf thru 0x030e66 (0xd388 bytes) 000587 1788747183.191210 mapread 0x018711 thru 0x01eae1 (0x63d1 bytes) 000588 1788747183.197921 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3a8d0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a8d0): Mirror 4: 0xfefd4580 (stale) Mirror 5: 0xfefd4580 (stale) Mirror 6: 0xfefd4580 (stale) Mirror 7: 0xfefd4580 (stale) Mirror 8: 0xfefd4580 (stale) Mirror 9: 0xc30052f3 (stale) Mirror 10: 0xc30052f3 (stale) CRC-32 checksum value for chunk [0, 0x3a8d0): Mirror 1: 0xfba26c3 Mirror 11: 0xfba26c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a8d0: skipped 000589 1788747184.103609 mapwrite 0x01cbfc thru 0x02abf9 (0xdffe bytes) 000590 1788747184.350660 mapread 0x0305c7 thru 0x03a8cf (0xa309 bytes) 000592 1788747184.384541 write 0x019f40 thru 0x0236e2 (0x97a3 bytes) 000593 1788747184.393389 write 0x032911 thru 0x03ffff (0xd6ef bytes) 000595 1788747184.397292 punch from 0x03af7d to 0x03ffff (0x5083 bytes) 000597 1788747184.437394 write 0x0089b5 thru 0x00bef3 (0x353f bytes) 000598 1788747184.455387 write 0x00ca8a thru 0x00fd5c (0x32d3 bytes) 000599 1788747184.501108 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x841dc691 (stale) Mirror 5: 0x841dc691 (stale) Mirror 6: 0x841dc691 (stale) Mirror 7: 0x841dc691 (stale) Mirror 8: 0x841dc691 (stale) Mirror 9: 0xa386412b (stale) Mirror 10: 0xa386412b (stale) Mirror 11: 0x420eb712 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb8cd06d Mirror 12: 0xb8cd06d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000600 1788747185.476920 mapwrite 0x0205f4 thru 0x02435d (0x3d6a bytes) 000601 1788747185.655311 mapread 0x03b704 thru 0x03ffff (0x48fc bytes) 000602 1788747185.667543 mapread 0x038e3f thru 0x03c1aa (0x336c bytes) 000605 1788747185.669447 mapwrite 0x036e7e thru 0x03ffff (0x9182 bytes) 000608 1788747185.732208 mapwrite 0x0380d8 thru 0x0398ff (0x1828 bytes) 000609 1788747185.755957 trunc from 0x040000 to 0x014948 (0x2b6b7 bytes) 000610 1788747185.787593 mapwrite 0x00bd99 thru 0x00c644 (0x8ac bytes) 000611 1788747185.822153 trunc from 0x014949 to 0x02f696 (0x1ad4e bytes) 000612 1788747185.834588 read 0x00afdf thru 0x00b0ca (0xec bytes) 000613 1788747185.838337 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2f697): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f697): Mirror 4: 0xf40bf025 (stale) Mirror 5: 0xf40bf025 (stale) Mirror 6: 0xf40bf025 (stale) Mirror 7: 0xf40bf025 (stale) Mirror 8: 0xf40bf025 (stale) Mirror 9: 0x957e020c (stale) Mirror 10: 0x957e020c (stale) Mirror 11: 0x33192f3c (stale) Mirror 12: 0x6b355978 (stale) CRC-32 checksum value for chunk [0, 0x2f697): Mirror 1: 0x19d53749 Mirror 13: 0x19d53749 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f697: skipped 000614 1788747186.594558 read 0x013f76 thru 0x015327 (0x13b2 bytes) 000615 1788747186.605594 trunc from 0x02f697 to 0x01d8bb (0x11ddb bytes) 000616 1788747186.745903 mapread 0x0188a3 thru 0x01d8bb (0x5019 bytes) 000617 1788747186.783365 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d8bc): Mirror 4: 0xde107bf2 (stale) Mirror 5: 0xde107bf2 (stale) Mirror 6: 0xde107bf2 (stale) Mirror 7: 0xde107bf2 (stale) Mirror 8: 0xde107bf2 (stale) Mirror 10: 0xde107bf2 (stale) Mirror 11: 0x11028f0 (stale) Mirror 12: 0x818a378d (stale) Mirror 13: 0x5da2cc7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d8bc): Mirror 4: 0xde107bf2 (stale) Mirror 5: 0xde107bf2 (stale) Mirror 6: 0xde107bf2 (stale) Mirror 7: 0xde107bf2 (stale) Mirror 8: 0xde107bf2 (stale) Mirror 10: 0xde107bf2 (stale) Mirror 11: 0x11028f0 (stale) Mirror 12: 0x818a378d (stale) Mirror 13: 0x5da2cc7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d8bc: skipped 000618 1788747187.433051 mapread 0x017cee thru 0x01d8bb (0x5bce bytes) 000621 1788747187.455658 mapwrite 0x00cb8e thru 0x01c163 (0xf5d6 bytes) 000622 1788747187.515502 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1d8bc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d8bc): Mirror 4: 0xde107bf2 (stale) Mirror 5: 0xde107bf2 (stale) Mirror 6: 0xde107bf2 (stale) Mirror 7: 0xde107bf2 (stale) Mirror 8: 0xde107bf2 (stale) Mirror 10: 0xde107bf2 (stale) Mirror 11: 0x11028f0 (stale) Mirror 12: 0x818a378d (stale) Mirror 13: 0x5da2cc7c (stale) CRC-32 checksum value for chunk [0, 0x1d8bc): Mirror 1: 0xaa5844fb Mirror 14: 0xaa5844fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d8bc: skipped 000624 1788747188.391600 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x1d8bc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d8bc): Mirror 4: 0xde107bf2 (stale) Mirror 5: 0xde107bf2 (stale) Mirror 6: 0xde107bf2 (stale) Mirror 7: 0xde107bf2 (stale) Mirror 8: 0xde107bf2 (stale) Mirror 10: 0xde107bf2 (stale) Mirror 11: 0x11028f0 (stale) Mirror 12: 0x818a378d (stale) Mirror 13: 0x5da2cc7c (stale) CRC-32 checksum value for chunk [0, 0x1d8bc): Mirror 1: 0xaa5844fb Mirror 14: 0xaa5844fb Mirror 15: 0xaa5844fb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d8bc: skipped 000626 1788747189.281965 write 0x001408 thru 0x0082c4 (0x6ebd bytes) 000628 1788747189.404166 mapread 0x005531 thru 0x012302 (0xcdd2 bytes) 000629 1788747189.433098 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d8bc): Mirror 4: 0xde107bf2 (stale) Mirror 5: 0xde107bf2 (stale) Mirror 6: 0xde107bf2 (stale) Mirror 7: 0xde107bf2 (stale) Mirror 8: 0xde107bf2 (stale) Mirror 10: 0xde107bf2 (stale) Mirror 11: 0x11028f0 (stale) Mirror 12: 0x818a378d (stale) Mirror 14: 0xaa5844fb (stale) Mirror 15: 0xaa5844fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d8bc): Mirror 4: 0xde107bf2 (stale) Mirror 5: 0xde107bf2 (stale) Mirror 6: 0xde107bf2 (stale) Mirror 7: 0xde107bf2 (stale) Mirror 8: 0xde107bf2 (stale) Mirror 10: 0xde107bf2 (stale) Mirror 11: 0x11028f0 (stale) Mirror 12: 0x818a378d (stale) Mirror 14: 0xaa5844fb (stale) Mirror 15: 0xaa5844fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d8bc: skipped 000630 1788747189.904736 trunc from 0x01d8bc to 0x03284b (0x14f90 bytes) 000631 1788747189.950298 MIRROR_RESYNC 11 mirrors 000632 1788747191.489060 mapread 0x027572 thru 0x03284b (0xb2da bytes) 000635 1788747191.521849 mapread 0x01b074 thru 0x027a12 (0xc99f bytes) 000637 1788747191.532255 write 0x01e74d thru 0x020a0d (0x22c1 bytes) 000638 1788747191.681882 write 0x037adf thru 0x03ffff (0x8521 bytes) 000639 1788747191.683145 trunc from 0x040000 to 0x03802f (0x7fd0 bytes) 000641 1788747191.711100 trunc from 0x038030 to 0x01633e (0x21cf1 bytes) 000642 1788747191.729300 punch from 0x0001f9 to 0x001cfb (0x1b03 bytes) 000643 1788747191.734459 trunc from 0x01633f to 0x014f7f (0x13bf bytes) 000644 1788747191.755328 trunc from 0x014f80 to 0x029df7 (0x14e78 bytes) 000645 1788747191.771434 mapwrite 0x01180d thru 0x01fc1a (0xe40e bytes) 000646 1788747191.854596 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29df8): Mirror 5: 0x30775133 (stale) Mirror 6: 0x30775133 (stale) Mirror 7: 0x30775133 (stale) Mirror 8: 0x30775133 (stale) Mirror 10: 0x30775133 (stale) Mirror 11: 0x30775133 (stale) Mirror 12: 0x30775133 (stale) Mirror 14: 0x30775133 (stale) Mirror 15: 0x30775133 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29df8): Mirror 5: 0x30775133 (stale) Mirror 6: 0x30775133 (stale) Mirror 7: 0x30775133 (stale) Mirror 8: 0x30775133 (stale) Mirror 10: 0x30775133 (stale) Mirror 11: 0x30775133 (stale) Mirror 12: 0x30775133 (stale) Mirror 14: 0x30775133 (stale) Mirror 15: 0x30775133 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29df8: skipped 000647 1788747192.327223 punch from 0x01390a to 0x021de3 (0xe4da bytes) 000648 1788747192.332062 write 0x02db8d thru 0x039a21 (0xbe95 bytes) 000649 1788747192.334842 punch from 0x00664f to 0x015b7e (0xf530 bytes) 000652 1788747192.347107 read 0x02b6b9 thru 0x034177 (0x8abf bytes) 000653 1788747192.352038 read 0x00a1df thru 0x00fc40 (0x5a62 bytes) 000654 1788747192.362024 mapread 0x027abc thru 0x0368c8 (0xee0d bytes) 000655 1788747192.363357 mapread 0x037bb4 thru 0x038235 (0x682 bytes) 000656 1788747192.368000 MIRROR_RESYNC 10 mirrors 000658 1788747193.590201 trunc from 0x039a22 to 0x03e710 (0x4cef bytes) 000659 1788747193.683045 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e711): Mirror 5: 0xd426ea15 (stale) Mirror 6: 0xd426ea15 (stale) Mirror 7: 0xd426ea15 (stale) Mirror 8: 0xd426ea15 (stale) Mirror 10: 0xd426ea15 (stale) Mirror 12: 0xd426ea15 (stale) Mirror 14: 0xd426ea15 (stale) Mirror 15: 0xd426ea15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e711): Mirror 5: 0xd426ea15 (stale) Mirror 6: 0xd426ea15 (stale) Mirror 7: 0xd426ea15 (stale) Mirror 8: 0xd426ea15 (stale) Mirror 10: 0xd426ea15 (stale) Mirror 12: 0xd426ea15 (stale) Mirror 14: 0xd426ea15 (stale) Mirror 15: 0xd426ea15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e711: skipped 000660 1788747194.174444 mapwrite 0x015d7b thru 0x01801b (0x22a1 bytes) 000661 1788747194.237961 mapwrite 0x00e875 thru 0x017d33 (0x94bf bytes) 000662 1788747194.281685 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3e711): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e711): Mirror 5: 0xd426ea15 (stale) Mirror 6: 0xd426ea15 (stale) Mirror 7: 0xd426ea15 (stale) Mirror 8: 0xd426ea15 (stale) Mirror 10: 0xd426ea15 (stale) Mirror 12: 0xd426ea15 (stale) Mirror 14: 0xd426ea15 (stale) Mirror 15: 0xd426ea15 (stale) CRC-32 checksum value for chunk [0, 0x3e711): Mirror 1: 0xd5add000 Mirror 16: 0xd5add000 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e711: skipped 000663 1788747195.066324 mapwrite 0x013130 thru 0x01390d (0x7de bytes) 000664 1788747195.207793 mapread 0x008df5 thru 0x009d40 (0xf4c bytes) 000665 1788747195.218986 write 0x00ef7f thru 0x018013 (0x9095 bytes) 000666 1788747195.224468 mapwrite 0x01a28f thru 0x01fda2 (0x5b14 bytes) 000669 1788747195.279968 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3e711): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e711): Mirror 5: 0xd426ea15 (stale) Mirror 6: 0xd426ea15 (stale) Mirror 7: 0xd426ea15 (stale) Mirror 8: 0xd426ea15 (stale) Mirror 10: 0xd426ea15 (stale) Mirror 12: 0xd426ea15 (stale) Mirror 14: 0xd426ea15 (stale) Mirror 15: 0xd426ea15 (stale) Mirror 16: 0xd5add000 (stale) CRC-32 checksum value for chunk [0, 0x3e711): Mirror 1: 0x85440414 Mirror 17: 0x85440414 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e711: skipped 000670 1788747196.082105 mapwrite 0x01010a thru 0x013e59 (0x3d50 bytes) 000671 1788747196.265793 trunc from 0x03e711 to 0x01c19d (0x22573 bytes) 000672 1788747196.313510 mapwrite 0x03eebd thru 0x03ffff (0x1143 bytes) 000673 1788747196.459911 trunc from 0x040000 to 0x02b4c8 (0x14b37 bytes) 000674 1788747196.486174 trunc from 0x02b4c9 to 0x0255da (0x5eee bytes) 000677 1788747196.511021 mapwrite 0x014290 thru 0x01dadf (0x9850 bytes) 000679 1788747196.577028 mapread 0x0068e3 thru 0x00b8f0 (0x500e bytes) 000680 1788747196.587051 mapwrite 0x02c114 thru 0x03279e (0x668b bytes) 000681 1788747196.706396 punch from 0x01300f to 0x019497 (0x6489 bytes) 000682 1788747196.716877 read 0x02e44f thru 0x03279e (0x4350 bytes) 000685 1788747196.718881 read 0x012b6b thru 0x019f1b (0x73b1 bytes) 000686 1788747196.732161 trunc from 0x03279f to 0x027198 (0xb606 bytes) 000687 1788747196.765930 write 0x01392e thru 0x01b01f (0x76f2 bytes) 000688 1788747196.768309 punch from 0x00a0b3 to 0x0127ce (0x871c bytes) 000689 1788747196.785282 read 0x011a8b thru 0x011e96 (0x40c bytes) 000691 1788747196.811306 trunc from 0x027199 to 0x00be46 (0x1b352 bytes) 000692 1788747196.857137 MIRROR_RESYNC 11 mirrors 000693 1788747198.049002 punch from 0x006db1 to 0x00bb62 (0x4db2 bytes) 000694 1788747198.131072 write 0x02466a thru 0x02688e (0x2225 bytes) 000695 1788747198.147301 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2688f): Mirror 5: 0x928bcddc (stale) Mirror 6: 0x928bcddc (stale) Mirror 7: 0x928bcddc (stale) Mirror 10: 0x928bcddc (stale) Mirror 12: 0x928bcddc (stale) Mirror 14: 0x928bcddc (stale) Mirror 15: 0x928bcddc (stale) Mirror 16: 0x928bcddc (stale) Mirror 17: 0x928bcddc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2688f): Mirror 5: 0x928bcddc (stale) Mirror 6: 0x928bcddc (stale) Mirror 7: 0x928bcddc (stale) Mirror 10: 0x928bcddc (stale) Mirror 12: 0x928bcddc (stale) Mirror 14: 0x928bcddc (stale) Mirror 15: 0x928bcddc (stale) Mirror 16: 0x928bcddc (stale) Mirror 17: 0x928bcddc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2688f: skipped 000696 1788747198.630914 read 0x0031c2 thru 0x00dd43 (0xab82 bytes) 000697 1788747198.647623 trunc from 0x02688f to 0x016fb4 (0xf8da bytes) 000698 1788747198.672131 MIRROR_RESYNC 10 mirrors 000699 1788747200.487465 MIRROR_RESYNC 10 mirrors 000700 1788747201.101327 read 0x00cdf1 thru 0x00eb68 (0x1d78 bytes) 000701 1788747201.129798 trunc from 0x016fb5 to 0x02dd50 (0x16d9c bytes) 000702 1788747201.368499 mapread 0x01cee9 thru 0x022dc3 (0x5edb bytes) 000704 1788747201.461240 trunc from 0x02dd51 to 0x01606f (0x17ce1 bytes) 000705 1788747201.539870 MIRROR_RESYNC 10 mirrors 000706 1788747203.584508 read 0x00ff76 thru 0x01606f (0x60fa bytes) 000707 1788747203.597525 MIRROR_RESYNC 10 mirrors 000708 1788747204.167194 punch from 0x0119a0 to 0x0126c8 (0xd29 bytes) 000710 1788747204.237034 read 0x01505f thru 0x01606f (0x1011 bytes) 000712 1788747204.243739 trunc from 0x016070 to 0x01e43c (0x83cd bytes) 000713 1788747204.271242 mapread 0x00abaf thru 0x017260 (0xc6b2 bytes) 000714 1788747204.325126 mapread 0x00dd9e thru 0x013b9a (0x5dfd bytes) 000715 1788747204.326240 read 0x00221e thru 0x00c93b (0xa71e bytes) 000716 1788747204.333504 trunc from 0x01e43d to 0x0299f6 (0xb5ba bytes) 000718 1788747204.349429 write 0x011860 thru 0x01f4b2 (0xdc53 bytes) 000719 1788747204.358181 mapread 0x01c056 thru 0x028516 (0xc4c1 bytes) 000720 1788747204.359123 mapread 0x0281c7 thru 0x0299f6 (0x1830 bytes) 000721 1788747204.359838 read 0x016c1c thru 0x01972a (0x2b0f bytes) 000722 1788747204.360855 read 0x0154d7 thru 0x0246db (0xf205 bytes) 000725 1788747204.362840 trunc from 0x0299f7 to 0x002dab (0x26c4b bytes) 000726 1788747204.382631 punch from 0x001466 to 0x002dab (0x1946 bytes) 000731 1788747204.389198 punch from 0x00101b to 0x002dab (0x1d91 bytes) 000732 1788747204.392668 mapread 0x001395 thru 0x002dab (0x1a17 bytes) 000733 1788747204.401860 read 0x001349 thru 0x002dab (0x1a63 bytes) 000734 1788747204.402834 mapwrite 0x01524d thru 0x01cf9f (0x7d53 bytes) 000735 1788747204.486119 read 0x008cba thru 0x00bed3 (0x321a bytes) 000736 1788747204.486948 write 0x032259 thru 0x037a8e (0x5836 bytes) 000737 1788747204.490219 write 0x00ca3a thru 0x00d475 (0xa3c bytes) 000738 1788747204.491402 trunc from 0x037a8f to 0x00549e (0x325f0 bytes) 000739 1788747204.512459 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x549f): Mirror 5: 0x3b82c01a (stale) Mirror 6: 0x3b82c01a (stale) Mirror 7: 0x3b82c01a (stale) Mirror 10: 0x3b82c01a (stale) Mirror 12: 0x3b82c01a (stale) Mirror 14: 0x3b82c01a (stale) Mirror 15: 0x3b82c01a (stale) Mirror 16: 0x3b82c01a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x549f): Mirror 5: 0x3b82c01a (stale) Mirror 6: 0x3b82c01a (stale) Mirror 7: 0x3b82c01a (stale) Mirror 10: 0x3b82c01a (stale) Mirror 12: 0x3b82c01a (stale) Mirror 14: 0x3b82c01a (stale) Mirror 15: 0x3b82c01a (stale) Mirror 16: 0x3b82c01a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x549f: skipped 000743 1788747204.879854 punch from 0x003457 to 0x00549e (0x2048 bytes) 000744 1788747204.892042 trunc from 0x00549f to 0x000476 (0x5028 bytes) 000746 1788747204.909429 write 0x01a563 thru 0x023d25 (0x97c3 bytes) 000747 1788747204.910696 read 0x006932 thru 0x00fdd8 (0x94a7 bytes) 000748 1788747204.918881 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x23d26): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23d26): Mirror 5: 0xd2c05db8 (stale) Mirror 6: 0xd2c05db8 (stale) Mirror 7: 0xd2c05db8 (stale) Mirror 10: 0xd2c05db8 (stale) Mirror 12: 0xd2c05db8 (stale) Mirror 14: 0xd2c05db8 (stale) Mirror 15: 0xd2c05db8 (stale) Mirror 16: 0xd2c05db8 (stale) CRC-32 checksum value for chunk [0, 0x23d26): Mirror 1: 0x40fb63d0 Mirror 17: 0x40fb63d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23d26: skipped 000750 1788747205.501254 punch from 0x00d0e2 to 0x011b1f (0x4a3e bytes) 000752 1788747205.569406 mapread 0x01385c thru 0x01eff2 (0xb797 bytes) 000755 1788747205.601058 punch from 0x015fed to 0x02331e (0xd332 bytes) 000757 1788747205.605881 punch from 0x010cfa to 0x012095 (0x139c bytes) 000760 1788747205.610420 punch from 0x020aba to 0x023d25 (0x326c bytes) 000761 1788747205.613864 punch from 0x01c893 to 0x023d25 (0x7493 bytes) 000762 1788747205.616686 write 0x02f4da thru 0x03e82c (0xf353 bytes) 000763 1788747205.620795 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3e82d): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e82d): Mirror 5: 0x4d60ca5d (stale) Mirror 6: 0x4d60ca5d (stale) Mirror 7: 0x4d60ca5d (stale) Mirror 10: 0x4d60ca5d (stale) Mirror 12: 0x4d60ca5d (stale) Mirror 14: 0x4d60ca5d (stale) Mirror 15: 0x4d60ca5d (stale) Mirror 16: 0x4d60ca5d (stale) Mirror 17: 0xf9b5e2b8 (stale) CRC-32 checksum value for chunk [0, 0x3e82d): Mirror 1: 0x77f59e79 Mirror 18: 0x77f59e79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e82d: skipped 000764 1788747206.705278 punch from 0x0061e5 to 0x00ba73 (0x588f bytes) 000768 1788747206.772593 read 0x035338 thru 0x03e82c (0x94f5 bytes) 000769 1788747206.786586 trunc from 0x03e82d to 0x01c583 (0x222a9 bytes) 000770 1788747206.823699 MIRROR_RESYNC 11 mirrors 000771 1788747208.982435 mapwrite 0x0015f7 thru 0x00ea24 (0xd42e bytes) 000772 1788747209.344363 punch from 0x0079c3 to 0x015bc5 (0xe203 bytes) 000773 1788747209.350339 read 0x009baa thru 0x00b516 (0x196d bytes) 000774 1788747209.357697 punch from 0x010816 to 0x018a7b (0x8266 bytes) 000775 1788747209.363392 mapread 0x015037 thru 0x01c583 (0x754d bytes) 000776 1788747209.375257 mapwrite 0x01edce thru 0x023a34 (0x4c67 bytes) 000777 1788747209.551248 trunc from 0x023a35 to 0x023d18 (0x2e4 bytes) 000778 1788747209.581691 write 0x02b72f thru 0x02c1c0 (0xa92 bytes) 000780 1788747209.585524 mapread 0x00f512 thru 0x010fec (0x1adb bytes) 000782 1788747209.587832 trunc from 0x02c1c1 to 0x01f0d0 (0xd0f0 bytes) 000783 1788747209.607109 read 0x014147 thru 0x01b3f4 (0x72ae bytes) 000788 1788747209.612378 mapwrite 0x005408 thru 0x009601 (0x41fa bytes) 000789 1788747209.736765 mapwrite 0x03b14c thru 0x03d22a (0x20df bytes) 000790 1788747209.884846 mapread 0x00127e thru 0x00b717 (0xa49a bytes) 000791 1788747209.885983 read 0x009a54 thru 0x015eff (0xc4ac bytes) 000792 1788747209.887175 punch from 0x0181a3 to 0x0219e6 (0x9844 bytes) 000794 1788747209.891662 write 0x03d29d thru 0x03ffff (0x2d63 bytes) 000795 1788747209.892952 trunc from 0x040000 to 0x035c95 (0xa36a bytes) 000796 1788747209.914097 trunc from 0x035c96 to 0x02b198 (0xaafd bytes) 000798 1788747209.941667 punch from 0x019399 to 0x02256f (0x91d7 bytes) 000800 1788747209.955711 mapwrite 0x01c389 thru 0x029910 (0xd588 bytes) 000801 1788747210.018655 mapwrite 0x001caa thru 0x003f62 (0x22b9 bytes) 000802 1788747210.059670 write 0x0058b4 thru 0x00e331 (0x8a7e bytes) 000803 1788747210.062119 mapwrite 0x014b85 thru 0x0231d6 (0xe652 bytes) 000808 1788747210.191009 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b199): Mirror 5: 0x75dfc94 (stale) Mirror 6: 0x75dfc94 (stale) Mirror 7: 0x75dfc94 (stale) Mirror 10: 0x75dfc94 (stale) Mirror 12: 0x75dfc94 (stale) Mirror 14: 0x75dfc94 (stale) Mirror 15: 0x75dfc94 (stale) Mirror 17: 0x75dfc94 (stale) Mirror 18: 0x75dfc94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b199): Mirror 5: 0x75dfc94 (stale) Mirror 6: 0x75dfc94 (stale) Mirror 7: 0x75dfc94 (stale) Mirror 10: 0x75dfc94 (stale) Mirror 12: 0x75dfc94 (stale) Mirror 14: 0x75dfc94 (stale) Mirror 15: 0x75dfc94 (stale) Mirror 17: 0x75dfc94 (stale) Mirror 18: 0x75dfc94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b199: skipped 000809 1788747210.882809 write 0x00282c thru 0x010bde (0xe3b3 bytes) 000810 1788747210.896997 read 0x00a865 thru 0x00c422 (0x1bbe bytes) 000811 1788747210.897869 write 0x032364 thru 0x03427d (0x1f1a bytes) 000813 1788747210.901607 read 0x0209f9 thru 0x025d4c (0x5354 bytes) 000815 1788747210.931293 write 0x0096db thru 0x013d3d (0xa663 bytes) 000816 1788747210.933637 read 0x028bde thru 0x03427d (0xb6a0 bytes) 000818 1788747210.936192 punch from 0x003434 to 0x0111d0 (0xdd9d bytes) 000819 1788747210.962760 read 0x0056ea thru 0x00bd97 (0x66ae bytes) 000820 1788747210.976884 write 0x024ebe thru 0x03007a (0xb1bd bytes) 000821 1788747210.985311 trunc from 0x03427e to 0x0354e0 (0x1263 bytes) 000822 1788747211.014831 punch from 0x02ffa5 to 0x0354e0 (0x553c bytes) 000823 1788747211.037389 write 0x00689c thru 0x008bb9 (0x231e bytes) 000824 1788747211.039679 mapread 0x00b8d8 thru 0x01034d (0x4a76 bytes) 000825 1788747211.041783 punch from 0x00f4dc to 0x014368 (0x4e8d bytes) 000826 1788747211.061386 write 0x027c7f thru 0x03705a (0xf3dc bytes) 000827 1788747211.071425 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3705b): Mirror 5: 0xb3620e51 (stale) Mirror 6: 0xb3620e51 (stale) Mirror 7: 0xb3620e51 (stale) Mirror 10: 0xb3620e51 (stale) Mirror 12: 0xb3620e51 (stale) Mirror 14: 0xb3620e51 (stale) Mirror 15: 0xb3620e51 (stale) Mirror 18: 0xb3620e51 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3705b): Mirror 5: 0xb3620e51 (stale) Mirror 6: 0xb3620e51 (stale) Mirror 7: 0xb3620e51 (stale) Mirror 10: 0xb3620e51 (stale) Mirror 12: 0xb3620e51 (stale) Mirror 14: 0xb3620e51 (stale) Mirror 15: 0xb3620e51 (stale) Mirror 18: 0xb3620e51 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3705b: skipped 000828 1788747211.767426 write 0x003a2b thru 0x0070d5 (0x36ab bytes) 000829 1788747211.787638 mapwrite 0x0213c1 thru 0x02bc86 (0xa8c6 bytes) 000830 1788747212.035332 write 0x027b6a thru 0x02f8db (0x7d72 bytes) 000831 1788747212.046158 trunc from 0x03705b to 0x038143 (0x10e9 bytes) 000832 1788747212.082613 punch from 0x0334a9 to 0x034ded (0x1945 bytes) 000833 1788747212.102622 punch from 0x00dee8 to 0x017ac5 (0x9bde bytes) 000834 1788747212.121077 MIRROR_RESYNC 9 mirrors 000835 1788747213.917609 trunc from 0x038144 to 0x012d30 (0x25413 bytes) 000836 1788747214.021620 write 0x03e7ab thru 0x03ffff (0x1855 bytes) 000840 1788747214.023999 mapwrite 0x0168d3 thru 0x017504 (0xc32 bytes) 000841 1788747214.118135 mapwrite 0x02aa58 thru 0x03744f (0xc9f8 bytes) 000843 1788747214.257347 mapread 0x02f95c thru 0x03d0e1 (0xd786 bytes) 000844 1788747214.266666 punch from 0x01efb8 to 0x02097f (0x19c8 bytes) 000845 1788747214.274259 mapread 0x01fbc6 thru 0x022bfe (0x3039 bytes) 000846 1788747214.292509 trunc from 0x040000 to 0x020759 (0x1f8a6 bytes) 000848 1788747214.400829 write 0x037fb3 thru 0x03a741 (0x278f bytes) 000851 1788747214.404839 trunc from 0x03a742 to 0x030253 (0xa4ee bytes) 000853 1788747214.438174 mapwrite 0x03dedd thru 0x03ffff (0x2123 bytes) 000854 1788747214.535511 mapread 0x001ca9 thru 0x00483f (0x2b97 bytes) 000856 1788747214.603731 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2ffc41f2 (stale) Mirror 6: 0x2ffc41f2 (stale) Mirror 7: 0x2ffc41f2 (stale) Mirror 10: 0x2ffc41f2 (stale) Mirror 12: 0x2ffc41f2 (stale) Mirror 14: 0x2ffc41f2 (stale) Mirror 15: 0x2ffc41f2 (stale) Mirror 18: 0x2ffc41f2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfc3234ce Mirror 19: 0xfc3234ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000857 1788747215.760026 read 0x01fe3f thru 0x02b0e7 (0xb2a9 bytes) 000858 1788747215.782580 write 0x01909f thru 0x01cbca (0x3b2c bytes) 000859 1788747216.002586 mapread 0x0208e9 thru 0x0249fc (0x4114 bytes) 000860 1788747216.045559 read 0x017ddd thru 0x0230e7 (0xb30b bytes) 000862 1788747216.095241 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2ffc41f2 (stale) Mirror 6: 0x2ffc41f2 (stale) Mirror 7: 0x2ffc41f2 (stale) Mirror 10: 0x2ffc41f2 (stale) Mirror 12: 0x2ffc41f2 (stale) Mirror 14: 0x2ffc41f2 (stale) Mirror 15: 0x2ffc41f2 (stale) Mirror 19: 0xfc3234ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2ffc41f2 (stale) Mirror 6: 0x2ffc41f2 (stale) Mirror 7: 0x2ffc41f2 (stale) Mirror 10: 0x2ffc41f2 (stale) Mirror 12: 0x2ffc41f2 (stale) Mirror 14: 0x2ffc41f2 (stale) Mirror 15: 0x2ffc41f2 (stale) Mirror 19: 0xfc3234ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000867 1788747217.005264 punch from 0x039ed8 to 0x03ffff (0x6128 bytes) 000868 1788747217.034174 read 0x00cf4c thru 0x01c8d2 (0xf987 bytes) 000869 1788747217.080565 punch from 0x033875 to 0x0339c5 (0x151 bytes) 000870 1788747217.089523 punch from 0x0361b0 to 0x03ffff (0x9e50 bytes) 000871 1788747217.104116 read 0x03b6b1 thru 0x03ffff (0x494f bytes) 000872 1788747217.139526 read 0x0372ed thru 0x03ffff (0x8d13 bytes) 000873 1788747217.145707 mapwrite 0x00fc39 thru 0x019986 (0x9d4e bytes) 000874 1788747217.291291 trunc from 0x040000 to 0x027f3a (0x180c5 bytes) 000875 1788747217.339025 trunc from 0x027f3b to 0x014f86 (0x12fb4 bytes) 000877 1788747217.364932 mapwrite 0x0240ac thru 0x02b19f (0x70f4 bytes) 000878 1788747217.490624 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b1a0): Mirror 5: 0x2513c435 (stale) Mirror 7: 0x2513c435 (stale) Mirror 10: 0x2513c435 (stale) Mirror 12: 0x2513c435 (stale) Mirror 14: 0x2513c435 (stale) Mirror 15: 0x2513c435 (stale) Mirror 19: 0xc32927f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1a0): Mirror 5: 0x2513c435 (stale) Mirror 7: 0x2513c435 (stale) Mirror 10: 0x2513c435 (stale) Mirror 12: 0x2513c435 (stale) Mirror 14: 0x2513c435 (stale) Mirror 15: 0x2513c435 (stale) Mirror 19: 0xc32927f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1a0: skipped 000879 1788747218.330267 write 0x0092f6 thru 0x0189ba (0xf6c5 bytes) 000880 1788747218.393810 read 0x024055 thru 0x02b19f (0x714b bytes) 000882 1788747218.418195 mapread 0x01d1d5 thru 0x01f036 (0x1e62 bytes) 000884 1788747218.445017 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2b1a0): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1a0): Mirror 5: 0x2513c435 (stale) Mirror 7: 0x2513c435 (stale) Mirror 10: 0x2513c435 (stale) Mirror 12: 0x2513c435 (stale) Mirror 14: 0x2513c435 (stale) Mirror 15: 0x2513c435 (stale) Mirror 19: 0xc32927f6 (stale) CRC-32 checksum value for chunk [0, 0x2b1a0): Mirror 1: 0xd854ffaf Mirror 20: 0xd854ffaf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1a0: skipped 000887 1788747220.369949 mapread 0x016b98 thru 0x01df87 (0x73f0 bytes) 000888 1788747220.404532 write 0x02da03 thru 0x03adad (0xd3ab bytes) 000889 1788747220.582543 mapread 0x0385ac thru 0x0392c4 (0xd19 bytes) 000890 1788747220.584335 punch from 0x02e852 to 0x03adad (0xc55c bytes) 000891 1788747220.686026 mapwrite 0x03fc36 thru 0x03ffff (0x3ca bytes) 000894 1788747220.805488 trunc from 0x040000 to 0x009935 (0x366ca bytes) 000895 1788747220.856964 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x9936): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9936): Mirror 5: 0xf5419bf5 (stale) Mirror 7: 0xf5419bf5 (stale) Mirror 10: 0xf5419bf5 (stale) Mirror 12: 0xf5419bf5 (stale) Mirror 14: 0xf5419bf5 (stale) Mirror 15: 0xf5419bf5 (stale) Mirror 19: 0xf5419bf5 (stale) Mirror 20: 0x1e9b255c (stale) CRC-32 checksum value for chunk [0, 0x9936): Mirror 1: 0x1e9b255c Mirror 21: 0x1e9b255c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9936: skipped 000900 1788747222.714505 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x9936): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9936): Mirror 5: 0xf5419bf5 (stale) Mirror 7: 0xf5419bf5 (stale) Mirror 10: 0xf5419bf5 (stale) Mirror 12: 0xf5419bf5 (stale) Mirror 14: 0xf5419bf5 (stale) Mirror 15: 0xf5419bf5 (stale) Mirror 19: 0xf5419bf5 (stale) Mirror 20: 0x1e9b255c (stale) CRC-32 checksum value for chunk [0, 0x9936): Mirror 1: 0x1e9b255c Mirror 21: 0x1e9b255c Mirror 22: 0x1e9b255c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9936: skipped 000902 1788747224.066004 mapwrite 0x00dddb thru 0x018429 (0xa64f bytes) 000903 1788747224.249459 trunc from 0x01842a to 0x037140 (0x1ed17 bytes) 000904 1788747224.271160 trunc from 0x037141 to 0x022a48 (0x146f8 bytes) 000905 1788747224.300700 punch from 0x00f7e9 to 0x0179dd (0x81f5 bytes) 000906 1788747224.308526 mapwrite 0x02d979 thru 0x035ab3 (0x813b bytes) 000907 1788747224.415631 trunc from 0x035ab4 to 0x017d8b (0x1dd28 bytes) 000908 1788747224.437332 trunc from 0x017d8c to 0x016a1b (0x1370 bytes) 000909 1788747224.462307 write 0x023567 thru 0x02491b (0x13b5 bytes) 000910 1788747224.468401 mapread 0x00b737 thru 0x01512f (0x99f9 bytes) 000911 1788747224.483369 mapwrite 0x0229c6 thru 0x02640f (0x3a4a bytes) 000912 1788747224.689141 read 0x02094f thru 0x02640f (0x5ac1 bytes) 000913 1788747224.689827 mapwrite 0x00e753 thru 0x011852 (0x3100 bytes) 000914 1788747224.756911 mapread 0x00a6fa thru 0x014fa0 (0xa8a7 bytes) 000915 1788747224.764805 MIRROR_RESYNC 11 mirrors 000916 1788747227.407287 read 0x016435 thru 0x01d91d (0x74e9 bytes) 000917 1788747227.413021 trunc from 0x026410 to 0x003136 (0x232d9 bytes) 000918 1788747227.572462 mapread 0x000d39 thru 0x001b0a (0xdd2 bytes) 000919 1788747227.587186 read 0x00034f thru 0x003136 (0x2de8 bytes) 000920 1788747227.602605 MIRROR_RESYNC 11 mirrors truncating to largest ever: 0x3fbc9 000921 1788747229.058697 trunc from 0x003137 to 0x03fbc8 (0x3ca92 bytes) 000922 1788747229.150540 mapread 0x02148d thru 0x026d2e (0x58a2 bytes) 000923 1788747229.189529 mapread 0x03dd6a thru 0x03fbc8 (0x1e5f bytes) 000924 1788747229.204944 write 0x009cc4 thru 0x00bd9c (0x20d9 bytes) 000925 1788747229.215027 mapread 0x006d3d thru 0x0141f6 (0xd4ba bytes) 000926 1788747229.230786 mapwrite 0x024cde thru 0x02d866 (0x8b89 bytes) 000927 1788747229.289609 trunc from 0x03fbc9 to 0x008c3f (0x36f89 bytes) 000928 1788747229.322012 mapwrite 0x03eac6 thru 0x03ffff (0x153a bytes) 000929 1788747229.363068 mapwrite 0x007b38 thru 0x012e46 (0xb30f bytes) 000932 1788747229.433631 trunc from 0x040000 to 0x02f59e (0x10a61 bytes) 000934 1788747229.479617 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2f59f): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f59f): Mirror 5: 0xfdd44f6e (stale) Mirror 7: 0xfdd44f6e (stale) Mirror 10: 0xfdd44f6e (stale) Mirror 12: 0xfdd44f6e (stale) Mirror 14: 0xfdd44f6e (stale) Mirror 15: 0xfdd44f6e (stale) Mirror 19: 0xfdd44f6e (stale) Mirror 20: 0xfdd44f6e (stale) Mirror 21: 0xfdd44f6e (stale) Mirror 22: 0xfdd44f6e (stale) CRC-32 checksum value for chunk [0, 0x2f59f): Mirror 1: 0x3bd75b90 Mirror 23: 0x3bd75b90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f59f: skipped 000935 1788747230.817165 write 0x002e01 thru 0x00c036 (0x9236 bytes) 000936 1788747230.903922 punch from 0x00d012 to 0x017849 (0xa838 bytes) 000938 1788747230.919092 punch from 0x00047b to 0x00d216 (0xcd9c bytes) 000939 1788747230.922974 read 0x00209f thru 0x00e366 (0xc2c8 bytes) 000940 1788747230.932991 trunc from 0x02f59f to 0x025da1 (0x97fd bytes) 000941 1788747230.949718 write 0x0162a2 thru 0x01b70c (0x546b bytes) 000942 1788747230.958571 punch from 0x00b0bc to 0x018f10 (0xde55 bytes) 000943 1788747230.970455 trunc from 0x025da2 to 0x024717 (0x168a bytes) 000944 1788747230.988012 write 0x0122ab thru 0x01a77f (0x84d5 bytes) 000947 1788747231.013009 write 0x00b8e9 thru 0x00fa48 (0x4160 bytes) 000948 1788747231.041008 read 0x009991 thru 0x01804e (0xe6be bytes) 000949 1788747231.069116 write 0x019b04 thru 0x0216bd (0x7bba bytes) 000950 1788747231.071292 write 0x006a1d thru 0x00caa2 (0x6086 bytes) 000951 1788747231.074419 read 0x01db75 thru 0x024717 (0x6ba3 bytes) 000952 1788747231.078046 mapwrite 0x00d223 thru 0x0194da (0xc2b8 bytes) 000953 1788747231.168470 mapread 0x011ad0 thru 0x0144e2 (0x2a13 bytes) 000954 1788747231.169320 read 0x004600 thru 0x00e9ac (0xa3ad bytes) 000955 1788747231.169860 write 0x034927 thru 0x03b447 (0x6b21 bytes) 000957 1788747231.172417 mapwrite 0x01bbf6 thru 0x01c5e2 (0x9ed bytes) 000958 1788747231.204077 read 0x008bea thru 0x00e68f (0x5aa6 bytes) 000959 1788747231.204859 trunc from 0x03b448 to 0x022559 (0x18eee bytes) 000960 1788747231.217627 mapread 0x00e810 thru 0x019d73 (0xb564 bytes) 000961 1788747231.218925 read 0x018240 thru 0x022559 (0xa31a bytes) 000963 1788747231.220591 mapwrite 0x01f80f thru 0x029f85 (0xa777 bytes) 000964 1788747231.287414 write 0x039961 thru 0x03ffff (0x669f bytes) 000965 1788747231.289146 read 0x01dadd thru 0x02198a (0x3eae bytes) 000967 1788747231.291115 write 0x01e0d4 thru 0x02929f (0xb1cc bytes) 000968 1788747231.293122 read 0x0175ad thru 0x0259d6 (0xe42a bytes) 000970 1788747231.294730 write 0x01b246 thru 0x01d33a (0x20f5 bytes) 000971 1788747231.296083 punch from 0x035a87 to 0x03ffff (0xa579 bytes) 000972 1788747231.314030 read 0x03ab3f thru 0x03ffff (0x54c1 bytes) 000973 1788747231.323566 trunc from 0x040000 to 0x00c693 (0x3396c bytes) 000974 1788747231.343309 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc694): Mirror 7: 0x6d9c7a9c (stale) Mirror 10: 0x6d9c7a9c (stale) Mirror 12: 0x6d9c7a9c (stale) Mirror 14: 0x6d9c7a9c (stale) Mirror 15: 0x6d9c7a9c (stale) Mirror 19: 0x6d9c7a9c (stale) Mirror 20: 0x6d9c7a9c (stale) Mirror 21: 0x6d9c7a9c (stale) Mirror 22: 0x6d9c7a9c (stale) Mirror 23: 0xdf94c743 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc694): Mirror 7: 0x6d9c7a9c (stale) Mirror 10: 0x6d9c7a9c (stale) Mirror 12: 0x6d9c7a9c (stale) Mirror 14: 0x6d9c7a9c (stale) Mirror 15: 0x6d9c7a9c (stale) Mirror 19: 0x6d9c7a9c (stale) Mirror 20: 0x6d9c7a9c (stale) Mirror 21: 0x6d9c7a9c (stale) Mirror 22: 0x6d9c7a9c (stale) Mirror 23: 0xdf94c743 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc694: skipped 000975 1788747231.788088 read 0x006158 thru 0x0099aa (0x3853 bytes) 000976 1788747231.794722 mapread 0x00b8bd thru 0x00c693 (0xdd7 bytes) 000977 1788747231.809139 mapread 0x00132f thru 0x00c693 (0xb365 bytes) 000978 1788747231.824597 mapread 0x000fbf thru 0x00798d (0x69cf bytes) 000979 1788747231.838919 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc694): Mirror 10: 0x6d9c7a9c (stale) Mirror 12: 0x6d9c7a9c (stale) Mirror 14: 0x6d9c7a9c (stale) Mirror 15: 0x6d9c7a9c (stale) Mirror 19: 0x6d9c7a9c (stale) Mirror 20: 0x6d9c7a9c (stale) Mirror 21: 0x6d9c7a9c (stale) Mirror 22: 0x6d9c7a9c (stale) Mirror 23: 0xdf94c743 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc694): Mirror 10: 0x6d9c7a9c (stale) Mirror 12: 0x6d9c7a9c (stale) Mirror 14: 0x6d9c7a9c (stale) Mirror 15: 0x6d9c7a9c (stale) Mirror 19: 0x6d9c7a9c (stale) Mirror 20: 0x6d9c7a9c (stale) Mirror 21: 0x6d9c7a9c (stale) Mirror 22: 0x6d9c7a9c (stale) Mirror 23: 0xdf94c743 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc694: skipped 000980 1788747232.647153 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc694): Mirror 10: 0x6d9c7a9c (stale) Mirror 12: 0x6d9c7a9c (stale) Mirror 14: 0x6d9c7a9c (stale) Mirror 15: 0x6d9c7a9c (stale) Mirror 19: 0x6d9c7a9c (stale) Mirror 20: 0x6d9c7a9c (stale) Mirror 21: 0x6d9c7a9c (stale) Mirror 22: 0x6d9c7a9c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc694): Mirror 10: 0x6d9c7a9c (stale) Mirror 12: 0x6d9c7a9c (stale) Mirror 14: 0x6d9c7a9c (stale) Mirror 15: 0x6d9c7a9c (stale) Mirror 19: 0x6d9c7a9c (stale) Mirror 20: 0x6d9c7a9c (stale) Mirror 21: 0x6d9c7a9c (stale) Mirror 22: 0x6d9c7a9c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc694: skipped 000981 1788747233.015390 punch from 0x0058ef to 0x00c693 (0x6da5 bytes) 000982 1788747233.031719 trunc from 0x00c694 to 0x013a9e (0x740b bytes) 000984 1788747233.082601 mapread 0x00c983 thru 0x00da27 (0x10a5 bytes) 000985 1788747233.109301 trunc from 0x013a9f to 0x02f508 (0x1ba6a bytes) 000986 1788747233.137022 mapwrite 0x03311c thru 0x0349f6 (0x18db bytes) 000988 1788747233.280951 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x349f7): Mirror 10: 0xed695ee (stale) Mirror 12: 0xed695ee (stale) Mirror 14: 0xed695ee (stale) Mirror 15: 0xed695ee (stale) Mirror 19: 0xed695ee (stale) Mirror 20: 0xed695ee (stale) Mirror 21: 0xed695ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349f7): Mirror 10: 0xed695ee (stale) Mirror 12: 0xed695ee (stale) Mirror 14: 0xed695ee (stale) Mirror 15: 0xed695ee (stale) Mirror 19: 0xed695ee (stale) Mirror 20: 0xed695ee (stale) Mirror 21: 0xed695ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349f7: skipped 000989 1788747234.010071 write 0x03f983 thru 0x03ffff (0x67d bytes) 000991 1788747234.013010 punch from 0x031742 to 0x03195a (0x219 bytes) 000992 1788747234.034192 punch from 0x034538 to 0x03aed8 (0x69a1 bytes) 000993 1788747234.043108 punch from 0x00e4b2 to 0x01db34 (0xf683 bytes) 000994 1788747234.054130 mapread 0x010367 thru 0x01487d (0x4517 bytes) 000995 1788747234.120611 read 0x0317e3 thru 0x032a89 (0x12a7 bytes) 000998 1788747234.136934 mapread 0x025a85 thru 0x02cc07 (0x7183 bytes) 001000 1788747234.162863 MIRROR_RESYNC 8 mirrors All operations completed A-OK! PASS 70a (100s) == sanity-flr test 71: check mirror extend parent fid ==== 22:14:05 (1788747245) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 22:14:10 (1788747250) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x325:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.320254 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x325:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124272 s, 8.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x326:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x325:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x326:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x325:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 22:14:19 (1788747259) Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre2 Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8208448 .. Extending file size to 7052192 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7167776 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7233728 .. Extending file size to 7272800 .. Extending file size to 8427872 .. Extending file size to 8479840 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8653888 .. failed Extending file size to 8921120 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8458016 .. Extending file size to 9119808 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57537 57539 57540 57543 57545 done 192.168.202.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg226-client.virtnet /mnt/lustre2 (opts:) 192.168.202.126@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg226-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:15:41 (1788747341) Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre2 Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3466016 .. Extending file size to 1279232 .. done Extending file size to 4891840 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6124192 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7271776 .. Extending file size to 5295424 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5646944 .. Extending file size to 7202560 .. Extending file size to 7405440 .. Extending file size to 7580640 .. Extending file size to 8110144 .. Extending file size to 8977760 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8776864 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9062432 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59094 59095 59098 59099 59101 done 192.168.202.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg226-client.virtnet /mnt/lustre2 (opts:) 192.168.202.126@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg226-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:17:02 (1788747422) Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113288 s, 18.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.433351 s, 4.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x54408980 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg226-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:17:21 (1788747441) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108378 s, 19.4 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 22:17:27 (1788747447) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x231:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x358:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:17:34 (1788747454) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.185743 s, 11.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x359:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:17:41 (1788747461) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163346 s, 12.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x340:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x33f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x340:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35c:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:17:47 (1788747467) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.354812 s, 5.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x341:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x341:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35e:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:18:20 (1788747500) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg226-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142602 s, 7.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x342:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x360:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x342:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x360:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x35f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x343:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:19:16 (1788747556) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg226-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x361:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x362:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.309618 s, 6.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x361:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x362:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x344:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x361:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x362:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x344:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:20:10 (1788747610) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg226-server mds-ost sync done. Creating new pool test_204f oleg226-server: Pool lustre.test_204f created Adding targets to pool oleg226-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg226-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147657 s, 14.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x363:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x345:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x363:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x345:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg226-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg226-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg226-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:21:32 (1788747692) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:21:40 (1788747700) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:21:47 (1788747707) oleg226-server: Pool lustre.test_206 created Adding targets to pool oleg226-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg226-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg226-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg226-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg226-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:22:12 (1788747732) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641936 s, 16.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599619 s, 17.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc1b6fcb0 Mirror 3: 0xc1b6fcb0 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:22:23 (1788747743) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:22:26 (1788747746) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:22:30 (1788747750) Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.187998 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2370000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1500000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1500000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1500000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd00000 Read this (4 bytes): BBBB 192.168.202.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg226-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:23:10 (1788747790) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0115156 s, 711 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:23:18 (1788747798) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323208 s, 32.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x256 lmm_fid: [0x200000404:0x256:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x256 lmm_fid: [0x200000404:0x256:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834518 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 22:23:20.000000000 -0400 Modify: 2026-09-06 22:23:20.000000000 -0400 Change: 2026-09-06 22:23:21.000000000 -0400 Birth: 2026-09-06 22:23:19.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:23:27 (1788747807) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:23:30 (1788747810) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.591166 s, 69.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0776806 s, 52.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 6 22:23 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x258:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 6 22:23 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:23:39 (1788747819) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:23:47 (1788747827) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1730 sec =========== 22:23:56 (1788747836) === sanity-flr: start cleanup 22:23:58 (1788747838) === === sanity-flr: finish cleanup 22:24:03 (1788747843) ===