-----============= acceptance-small: sanity-flr ============----- Tue Sep 29 20:14:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg237-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:14:47 (1790727287) === oleg237-client.virtnet: executing check_config_client /mnt/lustre oleg237-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg237-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ecc42d23800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ecc42d23800.idle_timeout=debug disable quota as required oleg237-server: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:15:25 (1790727325) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8437] == sanity-flr test 0a: lfs mirror create with -N option == 20:15:27 (1790727327) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:15:44 (1790727344) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:15:47 (1790727347) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:15:51 (1790727351) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:16:15 (1790727375) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:16:18 (1790727378) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 20:16:21 (1790727381) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ecc42d23800.stats=clear osc.lustre-OST0001-osc-ffff8ecc42d23800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:16:30 (1790727390) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:16:41 (1790727401) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:16:49 (1790727409) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:16:53 (1790727413) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:17:01 (1790727421) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 20:17:11 (1790727431) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:17:18 (1790727438) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:17:26 (1790727446) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:17:34 (1790727454) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:17:44 (1790727464) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:17:51 (1790727471) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.87458 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266767 s, 3.9 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:18:02 (1790727482) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.480576 s, 32.7 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:18:10 (1790727490) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:18:21 (1790727501) Stopping /mnt/lustre-ost1 (opts:) on oleg237-server oleg237-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ecc42d23800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ecc42d23800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg237-server: oleg237-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg237-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg237-server oleg237-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ecc42d23800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ecc42d23800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg237-server: oleg237-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg237-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (64s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:19:25 (1790727565) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.491706 s, 17.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg237-server oleg237-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ecc42d23800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ecc42d23800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg237-server: oleg237-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg237-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:20:13 (1790727613) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg237-server oleg237-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ecc42d23800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ecc42d23800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg237-server: oleg237-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg237-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg237-server oleg237-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ecc42d23800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ecc42d23800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg237-server: oleg237-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg237-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:21:33 (1790727693) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.47701 s, 9.6 MB/s -rw-r--r-- 1 root root 52428800 Sep 29 20:21 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.566s user 0m0.001s sys 0m0.193s Stopping /mnt/lustre-ost1 (opts:) on oleg237-server oleg237-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ecc42d23800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ecc42d23800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.132s user 0m0.000s sys 0m0.186s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg237-server: oleg237-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg237-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg237-server oleg237-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ecc42d23800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ecc42d23800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.582s user 0m0.000s sys 0m0.203s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg237-server: oleg237-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg237-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ecc42d23800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ecc42d23800.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:23:11 (1790727791) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:23:14 (1790727794) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:23:18 (1790727798) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 20:23:22 (1790727802) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 20:23:30 (1790727810) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.667687 s, 18.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:23:42 (1790727822) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483565 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287937 s, 36.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:23:50 (1790727830) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:23:57 (1790727837) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00158871 s, 645 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000921032 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00103716 s, 987 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000982811 s, 1.0 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 20:24:06 (1790727846) create test files with size 15729721 .. b1596669889bef51fdf114e2524add41 - 860c39a36068c5ab3847bfa5292d39ac - 0215d4e9fefaec1374f54d0167cfd985 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 20:24:28 (1790727868) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:24:44 (1790727884) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:24:51 (1790727891) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170784 s, 6.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328724 s, 31.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 20:24:59 (1790727899) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175504 s, 11.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.338486 s, 12.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.264895 s, 7.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 20:25:18 (1790727918) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 20:25:22 (1790727922) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:25:27 (1790727927) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372292 s, 28.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 20:25:35 (1790727935) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.320511 s, 9.8 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:25:47 (1790727947) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:25:56 (1790727956) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.317144 s, 33.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:26:08 (1790727968) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8ecc4892c000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.3403 s, 30.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg237-client.virtnet /mnt/lustre (opts:) Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8ecc693e1000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 20:26:26 (1790727986) error lfs split: create victim file failed: File exists PASS 44f (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:26:45 (1790728005) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (42s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:27:27 (1790728047) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:28:10 (1790728090) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 20:28:15 (1790728095) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.319368 s, 9.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0894892 s, 35.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb75767ef (nosync,stale) Mirror 2: 0xb75767ef (nosync,stale) ** verify mirror 2 doesn't change original checksum: 42fec5e3a7b8b1551a2f6480e4934d7e - mirror 1 checksum: f35d1b687d76935bf51f9a55218613e0 - mirror 2 checksum: 42fec5e3a7b8b1551a2f6480e4934d7e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-29T20:28:19-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-29T20:28:19-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xff:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 42fec5e3a7b8b1551a2f6480e4934d7e - mirror 1 checksum: f35d1b687d76935bf51f9a55218613e0 - mirror 2 checksum: f35d1b687d76935bf51f9a55218613e0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-29T20:28:19-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-29T20:28:19-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xff:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:28:29 (1790728109) Failing mds1 on oleg237-server Stopping /mnt/lustre-mds1 (opts:) on oleg237-server 20:28:36 (1790728116) shut down facet: mds1 facet_host: oleg237-server facet_failover_host: oleg237-server Failover mds1 to oleg237-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 20:29:02 (1790728142) targets are mounted 20:29:02 (1790728142) facet_failover done oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x8:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03526 s, 29.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000405 lmm_object_id: 0x8 lmm_fid: [0x240000405:0x8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d:0x0] } PASS 50A (52s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:29:21 (1790728161) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:29:25 (1790728165) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:29:28 (1790728168) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00536459 s, 3.1 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:29:37 (1790728177) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:29:41 (1790728181) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0070396 s, 582 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0493477 s, 83.0 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:29:49 (1790728189) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x67:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:30:14 (1790728214) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:30:44 (1790728244) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x91:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:31:10 (1790728270) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:31:36 (1790728296) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 20:31:43 (1790728303) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57078 mirror split pid 57079 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 57078 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 57079 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 20:32:51 (1790728371) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 8373 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1790728373.869480 write 0x015811 thru 0x01fe4a (0xa63a bytes) 000004 1790728373.886587 read 0x008167 thru 0x00df30 (0x5dca bytes) truncating to largest ever: 0x1f96d 000005 1790728373.904996 trunc from 0x01fe4b to 0x01f96c (0x4de bytes) 000007 1790728373.934437 mapread 0x009ac5 thru 0x00c099 (0x25d5 bytes) 000008 1790728373.935438 mapwrite 0x0334da thru 0x03cac1 (0x95e8 bytes) truncating to largest ever: 0x3b8f9 000009 1790728374.173934 trunc from 0x03cac2 to 0x03b8f8 (0x11c9 bytes) 000011 1790728374.199316 read 0x013fb5 thru 0x0237c0 (0xf80c bytes) 000017 1790728374.229408 write 0x01e3de thru 0x0224a1 (0x40c4 bytes) 000018 1790728374.231105 punch from 0x011eee to 0x019373 (0x7486 bytes) 000020 1790728374.261332 read 0x0274fb thru 0x034f7f (0xda85 bytes) 000022 1790728374.265423 punch from 0x01a8d3 to 0x02a202 (0xf930 bytes) 000024 1790728374.296040 MIRROR_EXTEND to 2 mirrors 000025 1790728375.269216 mapwrite 0x02df80 thru 0x033fb5 (0x6036 bytes) 000027 1790728375.622312 mapwrite 0x03bd0f thru 0x03ffff (0x42f1 bytes) 000029 1790728375.879666 mapwrite 0x03c867 thru 0x03cd2f (0x4c9 bytes) 000030 1790728375.946956 write 0x03c142 thru 0x03ffff (0x3ebe bytes) 000033 1790728375.963561 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000034 1790728376.223571 mapwrite 0x015983 thru 0x020ce4 (0xb362 bytes) 000036 1790728376.295531 read 0x02f29e thru 0x032742 (0x34a5 bytes) 000037 1790728376.302518 mapread 0x02c468 thru 0x02e988 (0x2521 bytes) 000039 1790728376.314953 mapwrite 0x038809 thru 0x03ffff (0x77f7 bytes) 000040 1790728376.364697 punch from 0x01281f to 0x01f253 (0xca35 bytes) 000041 1790728376.370433 read 0x0022d8 thru 0x0055de (0x3307 bytes) 000042 1790728376.381374 trunc from 0x040000 to 0x035ea8 (0xa157 bytes) 000043 1790728376.398355 read 0x00d668 thru 0x0135f5 (0x5f8e bytes) 000044 1790728376.403349 MIRROR_EXTEND to 2 mirrors 000046 1790728377.132325 punch from 0x01e5b0 to 0x02c0d1 (0xdb22 bytes) 000047 1790728377.216552 read 0x0166e9 thru 0x021774 (0xb08c bytes) 000049 1790728377.233796 mapwrite 0x01e204 thru 0x027184 (0x8f81 bytes) 000050 1790728377.297406 write 0x037243 thru 0x03acdd (0x3a9b bytes) 000051 1790728377.314235 MIRROR_RESYNC 2 mirrors 000053 1790728378.067171 punch from 0x02b67c to 0x0335c5 (0x7f4a bytes) 000054 1790728378.182415 mapread 0x01fc9f thru 0x02ac2d (0xaf8f bytes) 000055 1790728378.232666 mapread 0x002480 thru 0x007d5d (0x58de bytes) 000056 1790728378.280750 mapread 0x0144fa thru 0x01a23d (0x5d44 bytes) 000057 1790728378.311534 read 0x0318b2 thru 0x03acdd (0x942c bytes) 000059 1790728378.325283 mapwrite 0x03ce9d thru 0x03ffff (0x3163 bytes) 000061 1790728378.556181 punch from 0x013df8 to 0x01d066 (0x926f bytes) 000062 1790728378.565204 mapwrite 0x034420 thru 0x03f084 (0xac65 bytes) 000063 1790728378.689849 trunc from 0x040000 to 0x03b79a (0x4865 bytes) 000065 1790728378.750819 trunc from 0x03b79b to 0x0006fd (0x3b09d bytes) 000066 1790728378.804212 read 0x0005e4 thru 0x0006fd (0x11a bytes) 000067 1790728378.815637 trunc from 0x0006fe to 0x010e7f (0x10782 bytes) 000070 1790728378.872684 punch from 0x00a08b to 0x010e7f (0x6df5 bytes) 000072 1790728378.888452 punch from 0x00c84f to 0x010e7f (0x4631 bytes) 000073 1790728378.898373 mapread 0x004a1d thru 0x010e7f (0xc463 bytes) 000074 1790728378.949346 mapread 0x00a7e6 thru 0x010e7f (0x669a bytes) 000077 1790728378.970514 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000078 1790728379.252597 read 0x004d9d thru 0x009f0f (0x5173 bytes) 000080 1790728379.266224 punch from 0x00b30b to 0x010e7f (0x5b75 bytes) 000081 1790728379.272126 mapread 0x00aa95 thru 0x00fddb (0x5347 bytes) 000082 1790728379.293917 punch from 0x00a9b8 to 0x00fcc4 (0x530d bytes) 000085 1790728379.301390 mapwrite 0x035daf thru 0x03a2c4 (0x4516 bytes) 000086 1790728379.383634 punch from 0x0206d0 to 0x025a6d (0x539e bytes) 000088 1790728379.391331 mapread 0x0363e2 thru 0x037886 (0x14a5 bytes) 000089 1790728379.392700 read 0x00cc15 thru 0x01c024 (0xf410 bytes) 000091 1790728379.437706 mapwrite 0x0174f0 thru 0x022b71 (0xb682 bytes) 000092 1790728379.514558 trunc from 0x03a2c5 to 0x009f09 (0x303bb bytes) 000093 1790728379.539648 write 0x01c630 thru 0x01d495 (0xe66 bytes) 000094 1790728379.542038 mapread 0x015c54 thru 0x01d495 (0x7842 bytes) 000095 1790728379.627409 punch from 0x010438 to 0x01a297 (0x9e60 bytes) 000099 1790728379.668445 trunc from 0x01d496 to 0x032894 (0x153ff bytes) 000100 1790728379.695518 MIRROR_EXTEND to 2 mirrors 000101 1790728380.846466 write 0x0233bc thru 0x0272c9 (0x3f0e bytes) 000102 1790728380.930916 mapread 0x00e123 thru 0x01ba27 (0xd905 bytes) 000103 1790728380.975732 read 0x006853 thru 0x012be6 (0xc394 bytes) 000104 1790728380.988992 punch from 0x01a067 to 0x01f4aa (0x5444 bytes) 000106 1790728381.005848 mapwrite 0x0218d8 thru 0x02324b (0x1974 bytes) 000108 1790728381.064147 mapread 0x00c4f4 thru 0x011515 (0x5022 bytes) 000109 1790728381.065204 mapwrite 0x0251cd thru 0x02a14a (0x4f7e bytes) 000111 1790728381.104327 mapread 0x01c26b thru 0x0270f6 (0xae8c bytes) 000112 1790728381.105593 mapwrite 0x03c162 thru 0x03ffff (0x3e9e bytes) 000113 1790728381.181398 punch from 0x02989b to 0x038132 (0xe898 bytes) 000115 1790728381.187678 mapread 0x027e18 thru 0x031a6c (0x9c55 bytes) 000117 1790728381.203708 trunc from 0x040000 to 0x002636 (0x3d9c9 bytes) 000119 1790728381.237199 write 0x03b28c thru 0x03ffff (0x4d74 bytes) 000120 1790728381.250726 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000121 1790728381.489795 write 0x002f0c thru 0x00a64d (0x7742 bytes) 000122 1790728381.510452 punch from 0x0043de to 0x0058d3 (0x14f6 bytes) 000123 1790728381.525684 mapread 0x01738d thru 0x01920b (0x1e7f bytes) 000124 1790728381.551436 write 0x011914 thru 0x01d42e (0xbb1b bytes) 000125 1790728381.557292 read 0x03c0fe thru 0x03ffff (0x3f02 bytes) 000126 1790728381.566313 write 0x028dbd thru 0x037b20 (0xed64 bytes) 000128 1790728381.579046 mapread 0x03058f thru 0x03465c (0x40ce bytes) 000129 1790728381.580201 mapwrite 0x02a6f8 thru 0x0348d7 (0xa1e0 bytes) 000131 1790728381.656227 mapread 0x028434 thru 0x03130d (0x8eda bytes) 000133 1790728381.660445 trunc from 0x040000 to 0x022349 (0x1dcb6 bytes) 000134 1790728381.684466 punch from 0x010517 to 0x011b0d (0x15f7 bytes) 000135 1790728381.691130 punch from 0x0143af to 0x01b3bb (0x700d bytes) 000136 1790728381.700959 MIRROR_EXTEND to 2 mirrors 000137 1790728382.446540 write 0x0386ae thru 0x03ffff (0x7952 bytes) 000138 1790728382.505540 read 0x01975c thru 0x024d7c (0xb621 bytes) 000140 1790728382.514841 read 0x0104a3 thru 0x017e5d (0x79bb bytes) 000141 1790728382.526873 punch from 0x01c86f to 0x028770 (0xbf02 bytes) 000142 1790728382.540563 mapwrite 0x02720b thru 0x031a39 (0xa82f bytes) 000143 1790728382.598461 punch from 0x03e69a to 0x03ee7f (0x7e6 bytes) 000144 1790728382.605991 punch from 0x000fdf to 0x00e5b5 (0xd5d7 bytes) 000145 1790728382.610642 punch from 0x01a827 to 0x029b93 (0xf36d bytes) 000146 1790728382.627343 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000149 1790728382.818408 read 0x01c9e8 thru 0x01d270 (0x889 bytes) 000151 1790728382.828859 read 0x02cbc1 thru 0x03120e (0x464e bytes) 000152 1790728382.845696 trunc from 0x040000 to 0x00c022 (0x33fdd bytes) 000153 1790728382.864850 mapwrite 0x007ce2 thru 0x008fb3 (0x12d2 bytes) 000154 1790728382.913279 read 0x005f5f thru 0x00c022 (0x60c4 bytes) 000157 1790728382.916878 punch from 0x007af1 to 0x00c022 (0x4532 bytes) 000158 1790728382.922616 trunc from 0x00c023 to 0x03749e (0x2b47c bytes) 000159 1790728382.944246 read 0x029b35 thru 0x02e4b1 (0x497d bytes) 000160 1790728382.981057 mapwrite 0x03fb6a thru 0x03ffff (0x496 bytes) 000161 1790728383.059556 mapwrite 0x03236b thru 0x03b7c5 (0x945b bytes) 000162 1790728383.112600 punch from 0x00cf83 to 0x00d5e2 (0x660 bytes) 000163 1790728383.117484 punch from 0x031eeb to 0x03ffff (0xe115 bytes) 000164 1790728383.122413 write 0x01f756 thru 0x020279 (0xb24 bytes) 000165 1790728383.124139 mapwrite 0x013354 thru 0x0200e1 (0xcd8e bytes) 000166 1790728383.284785 MIRROR_EXTEND to 2 mirrors 000167 1790728384.046351 write 0x022ae2 thru 0x02d96d (0xae8c bytes) 000168 1790728384.175669 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000169 1790728384.373478 punch from 0x01198e to 0x014ddb (0x344e bytes) 000170 1790728384.379558 punch from 0x00dfdf to 0x0116b2 (0x36d4 bytes) 000171 1790728384.386573 write 0x012b83 thru 0x01e6f8 (0xbb76 bytes) 000172 1790728384.403606 mapread 0x03dd3a thru 0x03e0ed (0x3b4 bytes) 000173 1790728384.414359 mapwrite 0x01deaf thru 0x025fca (0x811c bytes) 000175 1790728384.486364 trunc from 0x040000 to 0x00d72e (0x328d1 bytes) 000176 1790728384.530935 punch from 0x008b9f to 0x00d72e (0x4b90 bytes) 000177 1790728384.537940 trunc from 0x00d72f to 0x00990c (0x3e22 bytes) 000178 1790728384.557764 mapwrite 0x029531 thru 0x034a68 (0xb538 bytes) 000184 1790728384.664463 punch from 0x00b5fd to 0x013df9 (0x87fd bytes) 000188 1790728384.677312 read 0x00003d thru 0x00956e (0x9532 bytes) 000191 1790728384.689404 mapread 0x004dc3 thru 0x00ba72 (0x6cb0 bytes) 000192 1790728384.702121 read 0x030700 thru 0x034a68 (0x4369 bytes) 000193 1790728384.703063 mapwrite 0x00bdb8 thru 0x00dbf6 (0x1e3f bytes) 000194 1790728384.754232 trunc from 0x034a69 to 0x02987c (0xb1ec bytes) 000195 1790728384.776044 write 0x00dd45 thru 0x00e892 (0xb4e bytes) 000196 1790728384.777132 punch from 0x00263f to 0x004fa4 (0x2966 bytes) 000198 1790728384.789551 trunc from 0x02987d to 0x01e835 (0xb047 bytes) 000200 1790728384.813650 read 0x004625 thru 0x010aa7 (0xc483 bytes) 000201 1790728384.832784 MIRROR_EXTEND to 2 mirrors 000202 1790728385.747929 punch from 0x00a500 to 0x00f4f4 (0x4ff5 bytes) 000205 1790728385.847563 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e836): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e836): Mirror 2: 0xec2bb108 (stale) CRC-32 checksum value for chunk [0, 0x1e836): Mirror 1: 0xa975e25e Mirror 3: 0xa975e25e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e836: skipped 000206 1790728386.978695 mapwrite 0x02c3e5 thru 0x032679 (0x6295 bytes) 000210 1790728387.277926 write 0x00fe57 thru 0x0153e4 (0x558e bytes) 000211 1790728387.293234 mapwrite 0x02d117 thru 0x031756 (0x4640 bytes) 000213 1790728387.358498 punch from 0x00925f to 0x011346 (0x80e8 bytes) 000214 1790728387.371509 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3267a): Mirror 3: 0x13d9cdd7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3267a): Mirror 3: 0x13d9cdd7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3267a: skipped 000215 1790728387.658599 mapread 0x01292a thru 0x014ba1 (0x2278 bytes) 000216 1790728387.690636 mapwrite 0x03c72b thru 0x03ffff (0x38d5 bytes) 000219 1790728387.777457 punch from 0x035f53 to 0x03b656 (0x5704 bytes) 000220 1790728387.782716 mapread 0x02f119 thru 0x03a9fc (0xb8e4 bytes) 000222 1790728387.805641 mapread 0x031721 thru 0x038b93 (0x7473 bytes) 000223 1790728387.810074 read 0x00d63b thru 0x01524c (0x7c12 bytes) 000225 1790728387.811820 read 0x02efea thru 0x034d97 (0x5dae bytes) 000226 1790728387.816966 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000227 1790728387.959519 read 0x0154cd thru 0x017909 (0x243d bytes) 000228 1790728387.964640 write 0x03f18f thru 0x03ffff (0xe71 bytes) 000230 1790728387.972629 mapwrite 0x01dbe0 thru 0x025013 (0x7434 bytes) 000231 1790728388.031541 write 0x0348f2 thru 0x0380b9 (0x37c8 bytes) 000233 1790728388.046417 mapwrite 0x0006b2 thru 0x00ecfd (0xe64c bytes) 000234 1790728388.106203 read 0x003577 thru 0x00c829 (0x92b3 bytes) 000235 1790728388.107392 punch from 0x03ad53 to 0x03ffff (0x52ad bytes) 000236 1790728388.114213 punch from 0x0258db to 0x034bd3 (0xf2f9 bytes) 000239 1790728388.122157 trunc from 0x040000 to 0x03331d (0xcce2 bytes) 000240 1790728388.139274 punch from 0x02655f to 0x02dfe4 (0x7a86 bytes) 000242 1790728388.156843 MIRROR_EXTEND to 2 mirrors 000243 1790728388.979974 punch from 0x02355b to 0x02a7df (0x7285 bytes) 000244 1790728389.048261 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000246 1790728389.204358 MIRROR_EXTEND to 2 mirrors 000247 1790728389.990492 read 0x01ac3c thru 0x028d17 (0xe0dc bytes) 000248 1790728390.008795 read 0x008214 thru 0x00bd5d (0x3b4a bytes) 000250 1790728390.037050 mapread 0x01a51f thru 0x02690f (0xc3f1 bytes) 000251 1790728390.055241 punch from 0x003224 to 0x006ec9 (0x3ca6 bytes) 000253 1790728390.141802 mapread 0x012c48 thru 0x01ed87 (0xc140 bytes) 000254 1790728390.269554 read 0x0252cd thru 0x0258e3 (0x617 bytes) 000255 1790728390.295358 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3331e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3331e): Mirror 2: 0x2ce017c3 (stale) CRC-32 checksum value for chunk [0, 0x3331e): Mirror 1: 0xd8e51b12 Mirror 3: 0xd8e51b12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3331e: skipped 000257 1790728391.518937 mapread 0x00df8b thru 0x00e9cb (0xa41 bytes) 000258 1790728391.538583 mapwrite 0x0048a0 thru 0x00dd4f (0x94b0 bytes) 000260 1790728391.729053 mapread 0x0161f5 thru 0x020b9c (0xa9a8 bytes) 000261 1790728391.751562 read 0x033124 thru 0x03331d (0x1fa bytes) 000263 1790728391.758168 mapread 0x02bf87 thru 0x03331d (0x7397 bytes) 000264 1790728391.768094 read 0x005fae thru 0x01357d (0xd5d0 bytes) 000265 1790728391.768736 punch from 0x01c0ed to 0x0208c8 (0x47dc bytes) 000266 1790728391.773629 read 0x016319 thru 0x01c7fc (0x64e4 bytes) 000267 1790728391.784287 write 0x001088 thru 0x002e41 (0x1dba bytes) 000268 1790728391.785541 write 0x037458 thru 0x03ffff (0x8ba8 bytes) 000271 1790728391.788234 read 0x01326b thru 0x01bc75 (0x8a0b bytes) 000273 1790728391.790097 mapread 0x0000a1 thru 0x005fd6 (0x5f36 bytes) 000274 1790728391.791106 mapread 0x001ff4 thru 0x00c3c2 (0xa3cf bytes) 000275 1790728391.792799 mapread 0x01c8e2 thru 0x0208de (0x3ffd bytes) 000276 1790728391.797771 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7bcb1d46 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7bcb1d46 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000277 1790728392.182367 write 0x033b13 thru 0x037e19 (0x4307 bytes) 000279 1790728392.216368 mapwrite 0x00ec17 thru 0x016137 (0x7521 bytes) 000280 1790728392.385857 mapwrite 0x01d00f thru 0x025f13 (0x8f05 bytes) 000281 1790728392.475308 mapread 0x004c74 thru 0x00b3ca (0x6757 bytes) 000283 1790728392.497480 trunc from 0x040000 to 0x035632 (0xa9cd bytes) 000284 1790728392.528247 write 0x01a947 thru 0x0206b3 (0x5d6d bytes) 000285 1790728392.529364 mapwrite 0x03c34a thru 0x03ffff (0x3cb6 bytes) 000286 1790728392.620148 punch from 0x015255 to 0x01e348 (0x90f4 bytes) 000288 1790728392.627615 write 0x005158 thru 0x00ec9d (0x9b46 bytes) 000289 1790728392.629500 punch from 0x0393a1 to 0x03ffff (0x6c5f bytes) 000290 1790728392.657090 trunc from 0x040000 to 0x003b43 (0x3c4bc bytes) 000292 1790728392.683802 MIRROR_RESYNC 2 mirrors 000293 1790728393.292187 MIRROR_EXTEND to 3 mirrors 000295 1790728393.846502 trunc from 0x003b44 to 0x01f072 (0x1b52f bytes) 000297 1790728393.948523 mapwrite 0x018b3f thru 0x01dd50 (0x5212 bytes) 000298 1790728394.014581 punch from 0x004aed to 0x005146 (0x65a bytes) 000299 1790728394.020112 trunc from 0x01f073 to 0x001804 (0x1d86e bytes) 000300 1790728394.043524 MIRROR_RESYNC 3 mirrors 000301 1790728394.779338 mapread 0x0013c6 thru 0x001804 (0x43f bytes) 000302 1790728394.792441 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000303 1790728395.094649 punch from 0x001673 to 0x001804 (0x192 bytes) 000308 1790728395.216620 mapwrite 0x009619 thru 0x01367f (0xa067 bytes) 000309 1790728395.334519 mapread 0x006200 thru 0x0112f3 (0xb0f4 bytes) 000310 1790728395.335726 trunc from 0x013680 to 0x002597 (0x110e8 bytes) 000311 1790728395.359811 read 0x0021d9 thru 0x002597 (0x3bf bytes) 000312 1790728395.367382 mapwrite 0x023227 thru 0x0248cb (0x16a5 bytes) 000313 1790728395.439544 punch from 0x0050a2 to 0x0097be (0x471d bytes) 000314 1790728395.450167 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x248cc): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x248cc): Mirror 3: 0x2c9a498e (stale) CRC-32 checksum value for chunk [0, 0x248cc): Mirror 1: 0x52e31a5d Mirror 4: 0x52e31a5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x248cc: skipped 000315 1790728396.276878 write 0x00575a thru 0x00e386 (0x8c2d bytes) 000316 1790728396.360360 MIRROR_RESYNC 3 mirrors 000317 1790728397.278984 MIRROR_RESYNC 3 mirrors 000319 1790728397.606576 mapwrite 0x00aeae thru 0x0197be (0xe911 bytes) 000320 1790728397.787564 MIRROR_RESYNC 3 mirrors 000321 1790728398.648704 mapread 0x0066c7 thru 0x007bed (0x1527 bytes) 000322 1790728398.666595 trunc from 0x0248cc to 0x00b51f (0x193ac bytes) 000324 1790728398.847423 read 0x0006ed thru 0x00928d (0x8ba1 bytes) 000325 1790728398.862694 MIRROR_RESYNC 3 mirrors 000329 1790728399.868632 MIRROR_EXTEND to 4 mirrors 000330 1790728400.566675 mapwrite 0x00d21c thru 0x0177b7 (0xa59c bytes) 000331 1790728400.769167 MIRROR_RESYNC 4 mirrors 000332 1790728402.219408 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000333 1790728402.727727 write 0x03770b thru 0x03ffff (0x88f5 bytes) 000336 1790728402.920961 MIRROR_RESYNC 3 mirrors 000337 1790728404.761269 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000338 1790728405.129402 read 0x00816d thru 0x00de0d (0x5ca1 bytes) 000339 1790728405.138441 read 0x01aa06 thru 0x023761 (0x8d5c bytes) 000340 1790728405.152652 punch from 0x016ab6 to 0x0230cf (0xc61a bytes) 000341 1790728405.240188 read 0x01a088 thru 0x01c35b (0x22d4 bytes) 000342 1790728405.262713 write 0x021795 thru 0x026c87 (0x54f3 bytes) 000343 1790728405.265446 write 0x0161e6 thru 0x021741 (0xb55c bytes) 000346 1790728405.271347 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xde808d1f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbba1f637 Mirror 4: 0xbba1f637 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000347 1790728405.954081 read 0x02e4f4 thru 0x02fda1 (0x18ae bytes) 000349 1790728405.962157 read 0x027361 thru 0x02ac4c (0x38ec bytes) 000350 1790728405.971924 punch from 0x023f9b to 0x032bc1 (0xec27 bytes) 000351 1790728406.040042 write 0x0248e5 thru 0x025d58 (0x1474 bytes) 000352 1790728406.048160 write 0x021d41 thru 0x022c51 (0xf11 bytes) 000353 1790728406.056483 punch from 0x00cb37 to 0x01a289 (0xd753 bytes) 000354 1790728406.069472 mapwrite 0x03c98b thru 0x03ffff (0x3675 bytes) 000355 1790728406.125426 mapwrite 0x031f88 thru 0x032bb1 (0xc2a bytes) 000357 1790728406.179317 MIRROR_RESYNC 3 mirrors 000358 1790728407.248238 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000359 1790728407.707173 MIRROR_EXTEND to 3 mirrors 000361 1790728408.401395 punch from 0x0100bb to 0x01d3e9 (0xd32f bytes) 000363 1790728408.513166 trunc from 0x040000 to 0x033502 (0xcafd bytes) 000365 1790728408.551796 write 0x02dd58 thru 0x0395a1 (0xb84a bytes) 000366 1790728408.561339 trunc from 0x0395a2 to 0x023b71 (0x15a30 bytes) 000367 1790728408.574125 trunc from 0x023b72 to 0x01b2f1 (0x8880 bytes) 000368 1790728408.585299 write 0x00983c thru 0x009e34 (0x5f9 bytes) 000369 1790728408.590589 mapwrite 0x02e7d7 thru 0x03795f (0x9189 bytes) 000370 1790728408.674225 MIRROR_RESYNC 3 mirrors 000372 1790728409.528125 MIRROR_RESYNC 3 mirrors 000374 1790728409.856260 mapread 0x024055 thru 0x02e9fd (0xa9a9 bytes) 000375 1790728409.879163 mapread 0x01cbe9 thru 0x01f705 (0x2b1d bytes) 000378 1790728409.901258 MIRROR_RESYNC 3 mirrors 000379 1790728410.331554 mapread 0x0207f6 thru 0x030594 (0xfd9f bytes) 000381 1790728410.363826 write 0x00a606 thru 0x019903 (0xf2fe bytes) 000383 1790728410.500003 punch from 0x023dd9 to 0x030edf (0xd107 bytes) 000384 1790728410.526048 read 0x036bde thru 0x03795f (0xd82 bytes) 000385 1790728410.539010 mapread 0x003d82 thru 0x00a6f1 (0x6970 bytes) 000386 1790728410.560448 write 0x02d9cd thru 0x03b902 (0xdf36 bytes) 000387 1790728410.581268 read 0x012cf3 thru 0x0205ee (0xd8fc bytes) 000389 1790728410.620273 punch from 0x020116 to 0x02a59f (0xa48a bytes) 000391 1790728410.637556 write 0x027370 thru 0x036945 (0xf5d6 bytes) 000393 1790728410.646180 write 0x00d5e1 thru 0x0175f2 (0xa012 bytes) 000394 1790728410.648174 trunc from 0x03b903 to 0x031372 (0xa590 bytes) 000397 1790728410.676350 mapread 0x0095db thru 0x00dc05 (0x462b bytes) 000398 1790728410.677298 read 0x00bc5c thru 0x015d5c (0xa101 bytes) 000399 1790728410.679352 write 0x01f85e thru 0x02ce3b (0xd5de bytes) 000403 1790728410.684935 trunc from 0x031373 to 0x00ab8c (0x267e6 bytes) 000406 1790728410.706209 mapread 0x00a177 thru 0x00ab8c (0xa16 bytes) 000409 1790728410.710316 write 0x00d6ab thru 0x0145a2 (0x6ef8 bytes) 000410 1790728410.718061 MIRROR_RESYNC 3 mirrors 000411 1790728411.639768 MIRROR_RESYNC 3 mirrors 000413 1790728411.908755 punch from 0x002ba0 to 0x005efd (0x335e bytes) 000414 1790728411.991456 mapread 0x0000dc thru 0x006f6b (0x6e90 bytes) 000415 1790728412.022255 write 0x01b9fb thru 0x02b9bc (0xffc2 bytes) 000416 1790728412.024669 write 0x00def4 thru 0x0183b1 (0xa4be bytes) 000417 1790728412.031933 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b9bd): Mirror 5: 0xcd08e3d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b9bd): Mirror 5: 0xcd08e3d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b9bd: skipped 000418 1790728412.307721 trunc from 0x02b9bd to 0x033216 (0x785a bytes) 000420 1790728412.333560 mapwrite 0x0298c6 thru 0x03362c (0x9d67 bytes) 000421 1790728412.435664 read 0x01cfc2 thru 0x01e2de (0x131d bytes) 000422 1790728412.446674 MIRROR_RESYNC 2 mirrors 000424 1790728413.167179 MIRROR_RESYNC 2 mirrors 000425 1790728413.806654 write 0x006c5e thru 0x0155ac (0xe94f bytes) 000426 1790728413.938411 mapwrite 0x023059 thru 0x029890 (0x6838 bytes) 000427 1790728414.088314 punch from 0x003c12 to 0x010450 (0xc83f bytes) 000429 1790728414.098367 write 0x00595b thru 0x00a577 (0x4c1d bytes) 000431 1790728414.124362 MIRROR_RESYNC 2 mirrors 000432 1790728414.802644 mapread 0x029265 thru 0x02b0b6 (0x1e52 bytes) 000433 1790728414.828117 MIRROR_EXTEND to 3 mirrors 000435 1790728415.563694 trunc from 0x03362d to 0x000349 (0x332e3 bytes) 000436 1790728415.678123 MIRROR_RESYNC 3 mirrors 000437 1790728416.310106 MIRROR_EXTEND to 4 mirrors 000438 1790728416.852589 write 0x014491 thru 0x01d10e (0x8c7e bytes) 000439 1790728416.922473 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1d10f): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d10f): Mirror 5: 0x599c1b5c (stale) Mirror 6: 0x599c1b5c (stale) Mirror 7: 0x599c1b5c (stale) CRC-32 checksum value for chunk [0, 0x1d10f): Mirror 1: 0xa64146b Mirror 8: 0xa64146b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d10f: skipped 000440 1790728417.801038 read 0x0111e7 thru 0x017395 (0x61af bytes) 000443 1790728417.844339 write 0x0029f8 thru 0x00b369 (0x8972 bytes) 000444 1790728417.984470 trunc from 0x01d10f to 0x030798 (0x1368a bytes) 000447 1790728418.026844 punch from 0x00c93b to 0x016d52 (0xa418 bytes) 000449 1790728418.041991 punch from 0x01cb9f to 0x01eca1 (0x2103 bytes) 000451 1790728418.052138 punch from 0x0067e8 to 0x014399 (0xdbb2 bytes) 000452 1790728418.062606 punch from 0x015b95 to 0x0255ce (0xfa3a bytes) 000454 1790728418.071642 mapread 0x010d7a thru 0x01dda8 (0xd02f bytes) 000455 1790728418.134578 read 0x00a723 thru 0x00efeb (0x48c9 bytes) 000456 1790728418.142105 read 0x02bb3a thru 0x030798 (0x4c5f bytes) 000458 1790728418.143287 punch from 0x027451 to 0x02f56d (0x811d bytes) 000459 1790728418.154572 read 0x012898 thru 0x020381 (0xdaea bytes) 000461 1790728418.166417 MIRROR_RESYNC 5 mirrors 000462 1790728419.593306 read 0x000e6d thru 0x00a1a8 (0x933c bytes) 000463 1790728419.605209 mapread 0x02504e thru 0x027fc8 (0x2f7b bytes) 000464 1790728419.623096 mapread 0x01b31d thru 0x028ccc (0xd9b0 bytes) 000465 1790728419.631998 trunc from 0x030799 to 0x03a3ba (0x9c22 bytes) 000466 1790728419.722311 read 0x01a5d2 thru 0x01fd4f (0x577e bytes) 000467 1790728419.734386 punch from 0x03503d to 0x03a3ba (0x537e bytes) 000468 1790728419.738784 write 0x016f73 thru 0x0237af (0xc83d bytes) 000470 1790728419.741512 read 0x00dfa2 thru 0x0123a7 (0x4406 bytes) 000471 1790728419.742507 write 0x032983 thru 0x03ac5e (0x82dc bytes) 000472 1790728419.747587 mapread 0x03311a thru 0x033f08 (0xdef bytes) 000473 1790728419.753124 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ac5f): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac5f): Mirror 5: 0x53d0b5b9 (stale) Mirror 6: 0x53d0b5b9 (stale) Mirror 7: 0x53d0b5b9 (stale) Mirror 8: 0x53d0b5b9 (stale) CRC-32 checksum value for chunk [0, 0x3ac5f): Mirror 1: 0xa2b1c451 Mirror 9: 0xa2b1c451 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac5f: skipped 000474 1790728420.598781 MIRROR_RESYNC 6 mirrors 000476 1790728421.681268 punch from 0x0322e9 to 0x03ac5e (0x8976 bytes) 000477 1790728421.760913 trunc from 0x03ac5f to 0x025686 (0x155d8 bytes) 000478 1790728421.785811 mapread 0x01a8b9 thru 0x021082 (0x67ca bytes) 000480 1790728421.815229 mapread 0x012bc4 thru 0x014403 (0x1840 bytes) 000481 1790728421.827714 write 0x00b9a0 thru 0x0164f1 (0xab52 bytes) 000482 1790728421.829540 write 0x01a0bf thru 0x01e79d (0x46df bytes) truncating to largest ever: 0x3db12 000483 1790728421.831483 trunc from 0x025687 to 0x03db11 (0x1848b bytes) 000484 1790728421.848497 mapread 0x016f0f thru 0x023962 (0xca54 bytes) 000485 1790728421.849570 mapread 0x001163 thru 0x007d53 (0x6bf1 bytes) 000486 1790728421.859303 trunc from 0x03db12 to 0x010659 (0x2d4b8 bytes) 000488 1790728421.878960 read 0x0087a2 thru 0x010659 (0x7eb8 bytes) 000490 1790728421.880548 mapread 0x0076f3 thru 0x00ca83 (0x5391 bytes) 000491 1790728421.881222 mapwrite 0x00f2dd thru 0x01776b (0x848f bytes) 000492 1790728421.992633 mapread 0x008c55 thru 0x01340c (0xa7b8 bytes) 000493 1790728421.994006 punch from 0x00e30f to 0x0141db (0x5ecd bytes) 000494 1790728422.001215 mapwrite 0x001f8f thru 0x003dd4 (0x1e46 bytes) 000496 1790728422.152547 mapread 0x012367 thru 0x01776b (0x5405 bytes) 000497 1790728422.181586 read 0x0016be thru 0x0051f1 (0x3b34 bytes) 000498 1790728422.182197 read 0x00f109 thru 0x01776b (0x8663 bytes) 000499 1790728422.182763 mapwrite 0x033bc4 thru 0x036d6e (0x31ab bytes) 000500 1790728422.369325 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d6f): Mirror 5: 0xb4b29494 (stale) Mirror 6: 0xb4b29494 (stale) Mirror 7: 0xb4b29494 (stale) Mirror 8: 0xb4b29494 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d6f): Mirror 5: 0xb4b29494 (stale) Mirror 6: 0xb4b29494 (stale) Mirror 7: 0xb4b29494 (stale) Mirror 8: 0xb4b29494 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d6f: skipped 000501 1790728422.884864 trunc from 0x036d6f to 0x03531c (0x1a52 bytes) 000502 1790728422.901762 mapread 0x017fae thru 0x0196c8 (0x171b bytes) 000503 1790728422.921479 write 0x00e033 thru 0x01bda6 (0xdd74 bytes) 000504 1790728422.928509 punch from 0x015fb6 to 0x024a28 (0xea73 bytes) 000510 1790728422.958109 trunc from 0x03531d to 0x014206 (0x21116 bytes) 000511 1790728422.991570 read 0x00fab3 thru 0x014206 (0x4754 bytes) 000512 1790728422.993007 punch from 0x01082f to 0x014206 (0x39d8 bytes) 000513 1790728422.999844 read 0x000840 thru 0x005a3b (0x51fc bytes) 000515 1790728423.012417 write 0x02fa2a thru 0x03047b (0xa52 bytes) 000516 1790728423.014459 mapread 0x002595 thru 0x0088b5 (0x6321 bytes) 000517 1790728423.015786 mapread 0x00a331 thru 0x00bcb2 (0x1982 bytes) 000519 1790728423.018111 mapread 0x00e387 thru 0x013e15 (0x5a8f bytes) 000521 1790728423.021467 trunc from 0x03047c to 0x016ac3 (0x199b8 bytes) 000522 1790728423.039795 write 0x009ba6 thru 0x00b515 (0x1970 bytes) 000523 1790728423.042825 punch from 0x00149f to 0x0066ef (0x5251 bytes) 000524 1790728423.053221 write 0x016466 thru 0x01ad16 (0x48b1 bytes) 000527 1790728423.065023 trunc from 0x01ad17 to 0x036355 (0x1b63f bytes) 000528 1790728423.094172 punch from 0x001e2e to 0x008c9d (0x6e70 bytes) 000530 1790728423.109506 read 0x02e48d thru 0x02e68e (0x202 bytes) 000531 1790728423.121816 read 0x035c34 thru 0x036355 (0x722 bytes) 000532 1790728423.122849 trunc from 0x036356 to 0x01a0a7 (0x1c2ae bytes) 000533 1790728423.142837 mapread 0x015a8c thru 0x01a0a7 (0x461c bytes) 000535 1790728423.151926 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a0a8): Mirror 5: 0x535d7b9d (stale) Mirror 6: 0x535d7b9d (stale) Mirror 7: 0x535d7b9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0a8): Mirror 5: 0x535d7b9d (stale) Mirror 6: 0x535d7b9d (stale) Mirror 7: 0x535d7b9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0a8: skipped 000537 1790728423.520522 trunc from 0x01a0a8 to 0x00f3e5 (0xacc2 bytes) 000538 1790728423.537946 write 0x03c138 thru 0x03ffff (0x3ec8 bytes) 000539 1790728423.540192 mapread 0x027b0f thru 0x02860d (0xaff bytes) 000541 1790728423.550218 write 0x023d38 thru 0x03083f (0xcb08 bytes) 000542 1790728423.560208 mapwrite 0x03e859 thru 0x03ffff (0x17a7 bytes) 000543 1790728423.606502 punch from 0x01c4fe to 0x01ce46 (0x949 bytes) 000544 1790728423.614082 mapwrite 0x0300df thru 0x03e11d (0xe03f bytes) 000546 1790728423.699158 mapread 0x01a494 thru 0x02712f (0xcc9c bytes) 000547 1790728423.711136 mapwrite 0x00ee81 thru 0x0189e4 (0x9b64 bytes) 000549 1790728423.775631 mapwrite 0x00269e thru 0x0074bf (0x4e22 bytes) 000550 1790728423.838034 read 0x037d73 thru 0x03a805 (0x2a93 bytes) 000551 1790728423.839128 write 0x03e182 thru 0x03ffff (0x1e7e bytes) 000553 1790728423.841483 mapwrite 0x022e06 thru 0x02935b (0x6556 bytes) 000554 1790728423.906459 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc30ab52c (stale) Mirror 7: 0xc30ab52c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc30ab52c (stale) Mirror 7: 0xc30ab52c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000555 1790728424.212444 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc30ab52c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc30ab52c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000557 1790728424.476500 write 0x027a68 thru 0x031597 (0x9b30 bytes) 000560 1790728424.498447 trunc from 0x040000 to 0x01d175 (0x22e8a bytes) 000563 1790728424.528188 punch from 0x00ea6d to 0x01bc1c (0xd1b0 bytes) 000564 1790728424.543863 write 0x00a2f2 thru 0x00c143 (0x1e52 bytes) 000569 1790728424.564420 mapwrite 0x00bfc5 thru 0x00c280 (0x2bc bytes) 000571 1790728424.601073 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1d176): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d176): Mirror 6: 0x2f8f9d57 (stale) CRC-32 checksum value for chunk [0, 0x1d176): Mirror 1: 0x195eb6dc Mirror 7: 0x195eb6dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d176: skipped 000572 1790728425.477141 write 0x014a4d thru 0x01e526 (0x9ada bytes) 000573 1790728425.628793 mapread 0x010c34 thru 0x01a074 (0x9441 bytes) 000574 1790728425.657554 mapwrite 0x01d788 thru 0x024ae5 (0x735e bytes) 000575 1790728425.788968 mapread 0x020e30 thru 0x024ae5 (0x3cb6 bytes) 000576 1790728425.791475 mapread 0x00ed75 thru 0x01cc0e (0xde9a bytes) 000577 1790728425.793307 punch from 0x01d15f to 0x024ae5 (0x7987 bytes) 000578 1790728425.802529 punch from 0x013adc to 0x01d4b2 (0x99d7 bytes) 000582 1790728425.809817 punch from 0x002e71 to 0x010d50 (0xdee0 bytes) 000584 1790728425.816179 read 0x022dfd thru 0x024ae5 (0x1ce9 bytes) 000585 1790728425.822961 read 0x01fa90 thru 0x01fe4f (0x3c0 bytes) 000586 1790728425.838851 mapwrite 0x03a914 thru 0x03ffff (0x56ec bytes) 000588 1790728425.923159 punch from 0x01ebf6 to 0x026cf3 (0x80fe bytes) 000590 1790728425.929063 punch from 0x010ab8 to 0x010df5 (0x33e bytes) 000592 1790728425.933854 punch from 0x01e6a7 to 0x0297ce (0xb128 bytes) 000593 1790728425.938854 mapwrite 0x022aa2 thru 0x02f28c (0xc7eb bytes) 000594 1790728426.181270 punch from 0x02564d to 0x02fa45 (0xa3f9 bytes) 000595 1790728426.195090 trunc from 0x040000 to 0x01a12a (0x25ed5 bytes) 000596 1790728426.263015 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a12b): Mirror 7: 0x6688bee9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a12b): Mirror 7: 0x6688bee9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a12b: skipped 000597 1790728426.754737 trunc from 0x01a12b to 0x024c9b (0xab71 bytes) 000598 1790728426.789832 trunc from 0x024c9c to 0x02c7b6 (0x7b1b bytes) 000599 1790728426.813473 trunc from 0x02c7b7 to 0x02fd5a (0x35a4 bytes) 000600 1790728426.833710 mapread 0x008f00 thru 0x017371 (0xe472 bytes) 000601 1790728426.888653 read 0x009888 thru 0x0193cd (0xfb46 bytes) 000603 1790728426.894878 mapwrite 0x02f95a thru 0x038474 (0x8b1b bytes) 000606 1790728427.025835 punch from 0x002665 to 0x00a1cf (0x7b6b bytes) 000607 1790728427.031570 read 0x001c23 thru 0x006a57 (0x4e35 bytes) 000608 1790728427.041205 trunc from 0x038475 to 0x0078c6 (0x30bae bytes) 000609 1790728427.080698 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) 000610 1790728427.208826 write 0x0003b2 thru 0x00aaa4 (0xa6f3 bytes) 000611 1790728427.212959 mapwrite 0x00d2b7 thru 0x01662f (0x9379 bytes) 000614 1790728427.339850 trunc from 0x016630 to 0x0361fe (0x1fbcf bytes) 000615 1790728427.360383 mapwrite 0x03ac89 thru 0x03ffff (0x5377 bytes) 000616 1790728427.459122 mapread 0x0037d5 thru 0x0094f7 (0x5d23 bytes) 000617 1790728427.460146 write 0x0060f6 thru 0x00ef0a (0x8e15 bytes) 000618 1790728427.461971 write 0x0296d3 thru 0x02f485 (0x5db3 bytes) 000620 1790728427.475131 mapwrite 0x0260b9 thru 0x0280f0 (0x2038 bytes) 000622 1790728427.532613 trunc from 0x040000 to 0x011bed (0x2e412 bytes) 000624 1790728427.556551 read 0x00438c thru 0x0116ab (0xd320 bytes) 000626 1790728427.559972 mapwrite 0x014c75 thru 0x014ffc (0x388 bytes) 000627 1790728427.616377 mapwrite 0x03f399 thru 0x03ffff (0xc67 bytes) 000628 1790728427.703366 trunc from 0x040000 to 0x0072a9 (0x38d56 bytes) 000629 1790728427.722762 mapwrite 0x03cfe9 thru 0x03ffff (0x3017 bytes) 000631 1790728427.800820 mapwrite 0x000b89 thru 0x00ee79 (0xe2f1 bytes) 000632 1790728427.930133 punch from 0x016d69 to 0x01ffa6 (0x923e bytes) 000633 1790728427.940985 punch from 0x017ffb to 0x01f42e (0x7434 bytes) 000635 1790728427.960049 punch from 0x004ae1 to 0x011499 (0xc9b9 bytes) 000639 1790728427.998558 mapread 0x007491 thru 0x00abda (0x374a bytes) 000640 1790728428.023525 mapwrite 0x0010f0 thru 0x007aea (0x69fb bytes) 000641 1790728428.098768 write 0x032e73 thru 0x0388dc (0x5a6a bytes) 000643 1790728428.107325 write 0x019d6d thru 0x0256a5 (0xb939 bytes) 000645 1790728428.120877 trunc from 0x040000 to 0x03867b (0x7984 bytes) 000647 1790728428.148695 write 0x0066d6 thru 0x009288 (0x2bb3 bytes) 000648 1790728428.151186 trunc from 0x03867c to 0x023e0e (0x1486d bytes) 000650 1790728428.170857 mapwrite 0x02f3cc thru 0x036594 (0x71c9 bytes) 000652 1790728428.262806 mapread 0x0035c8 thru 0x00d9cc (0xa405 bytes) 000653 1790728428.264329 mapwrite 0x01db5d thru 0x02237e (0x4822 bytes) 000654 1790728428.327148 mapwrite 0x023d61 thru 0x02fcf8 (0xbf98 bytes) 000656 1790728428.395490 punch from 0x02e6e9 to 0x036594 (0x7eac bytes) 000658 1790728428.402216 trunc from 0x036595 to 0x00bc85 (0x2a90f bytes) 000660 1790728428.425697 mapread 0x00aa97 thru 0x00bc85 (0x11ef bytes) 000663 1790728428.429117 trunc from 0x00bc86 to 0x008b2f (0x3156 bytes) 000665 1790728428.453980 write 0x003567 thru 0x00a8d1 (0x736b bytes) 000666 1790728428.455412 punch from 0x007446 to 0x00a8d1 (0x348c bytes) 000671 1790728428.471173 mapwrite 0x011d5e thru 0x01583d (0x3ae0 bytes) 000672 1790728428.545974 punch from 0x003289 to 0x011641 (0xe3b9 bytes) 000673 1790728428.551785 trunc from 0x01583e to 0x003e17 (0x11a26 bytes) 000674 1790728428.570714 MIRROR_EXTEND to 2 mirrors 000675 1790728429.303386 MIRROR_RESYNC 2 mirrors 000676 1790728429.689842 punch from 0x000dba to 0x003e17 (0x305e bytes) 000677 1790728429.770920 trunc from 0x003e18 to 0x0005db (0x383c bytes) 000678 1790728429.793358 trunc from 0x0005dc to 0x005a75 (0x549a bytes) 000680 1790728429.809145 write 0x03922a thru 0x03ffff (0x6dd6 bytes) 000683 1790728429.816236 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdb7797c8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfc32c281 Mirror 3: 0xfc32c281 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000685 1790728430.746759 trunc from 0x040000 to 0x00a76f (0x35890 bytes) 000686 1790728430.819961 mapwrite 0x00f8ba thru 0x01ac86 (0xb3cd bytes) 000687 1790728430.907523 punch from 0x0024d8 to 0x00f9fe (0xd527 bytes) 000689 1790728430.913968 punch from 0x018677 to 0x01ac86 (0x2610 bytes) 000690 1790728430.923182 mapread 0x0056c6 thru 0x00caf4 (0x742f bytes) 000691 1790728430.933334 mapread 0x00a9aa thru 0x015b3d (0xb194 bytes) 000692 1790728430.934506 mapread 0x0001c5 thru 0x0012a3 (0x10df bytes) 000693 1790728430.941478 trunc from 0x01ac87 to 0x0024fa (0x1878c bytes) 000694 1790728430.958684 read 0x0024b8 thru 0x0024fa (0x43 bytes) 000696 1790728430.961383 mapread 0x000256 thru 0x0024fa (0x22a5 bytes) 000700 1790728430.965356 mapwrite 0x02738f thru 0x02b22f (0x3ea1 bytes) 000701 1790728431.058270 punch from 0x0128f1 to 0x01d981 (0xb091 bytes) 000702 1790728431.063861 trunc from 0x02b230 to 0x03a061 (0xee32 bytes) 000703 1790728431.080628 read 0x029e08 thru 0x0371bc (0xd3b5 bytes) 000704 1790728431.095610 punch from 0x02b85a to 0x033c4d (0x83f4 bytes) 000705 1790728431.107100 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a062): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a062): Mirror 2: 0x5b08d9f4 (stale) Mirror 3: 0x6708f00 (stale) CRC-32 checksum value for chunk [0, 0x3a062): Mirror 1: 0xccacc1d9 Mirror 4: 0xccacc1d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a062: skipped 000708 1790728432.184153 punch from 0x024589 to 0x02a3f2 (0x5e6a bytes) 000710 1790728432.263511 mapread 0x00e1d3 thru 0x00fd00 (0x1b2e bytes) 000712 1790728432.274070 write 0x03a51c thru 0x03ffff (0x5ae4 bytes) 000714 1790728432.283028 mapread 0x01d27e thru 0x025e83 (0x8c06 bytes) 000716 1790728432.312802 MIRROR_RESYNC 4 mirrors 000717 1790728433.792236 read 0x003b1f thru 0x00e132 (0xa614 bytes) 000718 1790728433.802825 trunc from 0x040000 to 0x009fa0 (0x3605f bytes) 000719 1790728433.885799 punch from 0x006944 to 0x009fa0 (0x365d bytes) 000720 1790728433.892801 read 0x00266d thru 0x007243 (0x4bd7 bytes) 000721 1790728433.902067 mapread 0x000166 thru 0x009fa0 (0x9e3b bytes) 000722 1790728433.903807 read 0x0065f7 thru 0x009fa0 (0x39aa bytes) 000724 1790728433.905483 mapwrite 0x0358f5 thru 0x03e8eb (0x8ff7 bytes) 000725 1790728433.993111 write 0x02f533 thru 0x03b138 (0xbc06 bytes) 000727 1790728433.996905 read 0x01c064 thru 0x026554 (0xa4f1 bytes) 000728 1790728433.997930 mapread 0x00cebe thru 0x0187f2 (0xb935 bytes) 000729 1790728434.004094 MIRROR_RESYNC 4 mirrors 000730 1790728435.261743 punch from 0x02131e to 0x0214ea (0x1cd bytes) 000731 1790728435.356728 read 0x021033 thru 0x025ff6 (0x4fc4 bytes) 000735 1790728435.370758 trunc from 0x03e8ec to 0x02f97c (0xef6f bytes) 000736 1790728435.401704 read 0x00347f thru 0x004d66 (0x18e8 bytes) 000737 1790728435.411220 punch from 0x0182ed to 0x021a9e (0x97b2 bytes) 000738 1790728435.417033 mapread 0x02c1f9 thru 0x02f97c (0x3784 bytes) 000741 1790728435.420008 write 0x0179e4 thru 0x017cf7 (0x314 bytes) 000742 1790728435.427266 MIRROR_RESYNC 4 mirrors 000743 1790728436.441688 MIRROR_EXTEND to 5 mirrors 000744 1790728437.225710 read 0x01741c thru 0x01ed03 (0x78e8 bytes) 000745 1790728437.238003 read 0x01187a thru 0x01ffad (0xe734 bytes) 000746 1790728437.253432 MIRROR_RESYNC 5 mirrors 000747 1790728437.710188 punch from 0x0134e6 to 0x01a8cf (0x73ea bytes) 000748 1790728437.816310 read 0x027bb8 thru 0x02e8ff (0x6d48 bytes) 000749 1790728437.830681 trunc from 0x02f97d to 0x009fb2 (0x259ca bytes) 000750 1790728437.858415 mapread 0x001518 thru 0x00449e (0x2f87 bytes) 000751 1790728437.881793 write 0x01f73a thru 0x023708 (0x3fcf bytes) 000752 1790728437.883690 mapread 0x01ef85 thru 0x022205 (0x3281 bytes) 000754 1790728437.892673 mapread 0x0233eb thru 0x023708 (0x31e bytes) 000755 1790728437.897822 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23709): Mirror 3: 0x8ee0efd2 (stale) Mirror 4: 0x8ee0efd2 (stale) Mirror 5: 0x8ee0efd2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23709): Mirror 3: 0x8ee0efd2 (stale) Mirror 4: 0x8ee0efd2 (stale) Mirror 5: 0x8ee0efd2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23709: skipped 000758 1790728438.293797 punch from 0x015ca4 to 0x023708 (0xda65 bytes) 000759 1790728438.302164 mapwrite 0x038423 thru 0x03ff58 (0x7b36 bytes) 000760 1790728438.505069 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ff59): Mirror 4: 0x3c7f1c71 (stale) Mirror 5: 0x3c7f1c71 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff59): Mirror 4: 0x3c7f1c71 (stale) Mirror 5: 0x3c7f1c71 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff59: skipped 000764 1790728438.901459 trunc from 0x03ff59 to 0x027a99 (0x184bf bytes) 000765 1790728438.937804 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27a9a): Mirror 5: 0xab51e729 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a9a): Mirror 5: 0xab51e729 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a9a: skipped 000766 1790728439.237607 read 0x000b2b thru 0x00c091 (0xb567 bytes) 000767 1790728439.247680 mapread 0x02044f thru 0x027a99 (0x764b bytes) 000768 1790728439.273375 mapwrite 0x004139 thru 0x01240d (0xe2d5 bytes) 000769 1790728439.338937 read 0x0071d3 thru 0x00fb7a (0x89a8 bytes) 000770 1790728439.340304 read 0x013ffe thru 0x01f027 (0xb02a bytes) 000774 1790728439.357600 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27a9a): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a9a): Mirror 5: 0xab51e729 (stale) CRC-32 checksum value for chunk [0, 0x27a9a): Mirror 1: 0x730b96a0 Mirror 6: 0x730b96a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a9a: skipped 000775 1790728440.175828 trunc from 0x027a9a to 0x00f72b (0x1836e bytes) 000776 1790728440.274966 mapread 0x003b77 thru 0x00eaa6 (0xaf30 bytes) 000777 1790728440.329826 read 0x004f77 thru 0x009122 (0x41ac bytes) 000778 1790728440.331584 mapwrite 0x03d935 thru 0x03ffff (0x26cb bytes) 000779 1790728440.492028 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb44b6fac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb44b6fac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000780 1790728440.936767 mapread 0x02822e thru 0x037279 (0xf04c bytes) 000782 1790728440.989647 write 0x01a5e4 thru 0x021027 (0x6a44 bytes) 000783 1790728441.004013 trunc from 0x040000 to 0x03785e (0x87a1 bytes) 000784 1790728441.023777 write 0x02f69a thru 0x03497c (0x52e3 bytes) 000787 1790728441.027687 mapread 0x030313 thru 0x03341a (0x3108 bytes) 000788 1790728441.029126 punch from 0x0337a5 to 0x03785e (0x40ba bytes) 000789 1790728441.041937 mapread 0x019667 thru 0x01d519 (0x3eb3 bytes) 000790 1790728441.052052 punch from 0x008d59 to 0x013cc7 (0xaf6f bytes) 000791 1790728441.060120 punch from 0x012a82 to 0x01d491 (0xaa10 bytes) 000793 1790728441.066309 punch from 0x005dea to 0x009677 (0x388e bytes) 000794 1790728441.071213 mapread 0x00e036 thru 0x019f61 (0xbf2c bytes) 000795 1790728441.094584 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3785f): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3785f): Mirror 6: 0xf790b2b2 (stale) CRC-32 checksum value for chunk [0, 0x3785f): Mirror 1: 0x18b42f84 Mirror 7: 0x18b42f84 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3785f: skipped 000796 1790728442.030647 mapread 0x033488 thru 0x03785e (0x43d7 bytes) 000797 1790728442.099732 mapread 0x00a45b thru 0x019714 (0xf2ba bytes) 000798 1790728442.142802 write 0x013973 thru 0x0194a5 (0x5b33 bytes) 000799 1790728442.321279 trunc from 0x03785f to 0x031b32 (0x5d2c bytes) 000800 1790728442.412868 mapwrite 0x0232ab thru 0x0243ae (0x1104 bytes) 000802 1790728442.506443 punch from 0x02afa1 to 0x02cc90 (0x1cf0 bytes) 000803 1790728442.518177 mapwrite 0x026bef thru 0x029a67 (0x2e79 bytes) 000806 1790728442.586932 trunc from 0x031b33 to 0x03c458 (0xa926 bytes) 000809 1790728442.613733 punch from 0x036f7c to 0x03c458 (0x54dd bytes) 000810 1790728442.619441 mapwrite 0x01d110 thru 0x029848 (0xc739 bytes) 000811 1790728442.664197 trunc from 0x03c459 to 0x025c06 (0x16852 bytes) 000812 1790728442.682485 MIRROR_RESYNC 3 mirrors 000813 1790728443.779790 read 0x01d99c thru 0x025c06 (0x826b bytes) 000814 1790728443.799025 write 0x028d35 thru 0x029004 (0x2d0 bytes) 000815 1790728443.925594 read 0x00638a thru 0x011413 (0xb08a bytes) 000817 1790728443.977947 write 0x02cfcb thru 0x037d21 (0xad57 bytes) 000820 1790728443.983112 write 0x03f676 thru 0x03ffff (0x98a bytes) 000821 1790728443.984958 mapread 0x028a83 thru 0x028f68 (0x4e6 bytes) 000823 1790728443.987769 trunc from 0x040000 to 0x0233f0 (0x1cc0f bytes) truncating to largest ever: 0x3f284 000824 1790728444.012803 trunc from 0x0233f1 to 0x03f283 (0x1be93 bytes) 000825 1790728444.057856 punch from 0x016878 to 0x017bce (0x1357 bytes) 000828 1790728444.072398 mapwrite 0x00ef27 thru 0x01c87d (0xd957 bytes) 000829 1790728444.235146 mapwrite 0x0200c7 thru 0x022365 (0x229f bytes) 000830 1790728444.290864 mapwrite 0x030e09 thru 0x03378c (0x2984 bytes) 000831 1790728444.365329 punch from 0x0274e0 to 0x030d7b (0x989c bytes) 000832 1790728444.377991 mapwrite 0x0277c7 thru 0x03513d (0xd977 bytes) 000833 1790728444.672963 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f284): Mirror 6: 0x862293c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f284): Mirror 6: 0x862293c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f284: skipped 000835 1790728444.971871 trunc from 0x03f284 to 0x02aaff (0x14784 bytes) 000837 1790728445.006339 mapwrite 0x029eb6 thru 0x033011 (0x915c bytes) 000839 1790728445.137830 write 0x002176 thru 0x003176 (0x1001 bytes) 000841 1790728445.156013 MIRROR_RESYNC 2 mirrors 000843 1790728446.084658 punch from 0x02ef08 to 0x033011 (0x410a bytes) 000844 1790728446.163048 punch from 0x023b8f to 0x0267ca (0x2c3c bytes) 000845 1790728446.167859 write 0x019c82 thru 0x01b192 (0x1511 bytes) 000846 1790728446.177220 mapread 0x005de4 thru 0x009820 (0x3a3d bytes) 000847 1790728446.199650 mapwrite 0x0037f4 thru 0x00d85a (0xa067 bytes) 000849 1790728446.249801 mapread 0x02e62e thru 0x02f124 (0xaf7 bytes) 000851 1790728446.260380 mapread 0x010917 thru 0x01a868 (0x9f52 bytes) 000854 1790728446.272200 punch from 0x00513e to 0x0091b3 (0x4076 bytes) 000855 1790728446.278042 read 0x0042c2 thru 0x00c788 (0x84c7 bytes) 000856 1790728446.287059 write 0x01ce5b thru 0x02c8e5 (0xfa8b bytes) 000857 1790728446.295712 trunc from 0x033012 to 0x0038a5 (0x2f76c bytes) 000858 1790728446.357200 read 0x00194e thru 0x0038a5 (0x1f58 bytes) 000859 1790728446.361931 read 0x00304f thru 0x0038a5 (0x857 bytes) 000860 1790728446.368645 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38a6): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a6): Mirror 6: 0x991c6ed (stale) CRC-32 checksum value for chunk [0, 0x38a6): Mirror 1: 0xe9595714 Mirror 7: 0xe9595714 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a6: skipped 000862 1790728447.364675 mapwrite 0x02f2cd thru 0x033912 (0x4646 bytes) 000863 1790728447.584982 write 0x00fb0d thru 0x01a7f3 (0xace7 bytes) 000865 1790728447.610512 punch from 0x012b94 to 0x019553 (0x69c0 bytes) 000866 1790728447.680868 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33913): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33913): Mirror 6: 0x344db017 (stale) Mirror 7: 0xd180763a (stale) CRC-32 checksum value for chunk [0, 0x33913): Mirror 1: 0x429b12 Mirror 8: 0x429b12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33913: skipped 000867 1790728448.961987 read 0x018ca6 thru 0x027c16 (0xef71 bytes) 000868 1790728448.986302 mapread 0x032473 thru 0x033912 (0x14a0 bytes) 000869 1790728449.051747 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33913): Mirror 6: 0x344db017 (stale) Mirror 7: 0xd180763a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33913): Mirror 6: 0x344db017 (stale) Mirror 7: 0xd180763a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33913: skipped 000870 1790728449.522973 mapread 0x00fbda thru 0x0129d3 (0x2dfa bytes) 000871 1790728449.562348 punch from 0x016e67 to 0x021d0d (0xaea7 bytes) 000873 1790728449.580167 mapread 0x029755 thru 0x033912 (0xa1be bytes) 000874 1790728449.595758 mapwrite 0x01cd5b thru 0x024184 (0x742a bytes) 000876 1790728449.665759 mapread 0x00a6b9 thru 0x00ef49 (0x4891 bytes) 000877 1790728449.681720 read 0x032604 thru 0x033912 (0x130f bytes) 000878 1790728449.687193 read 0x0312b2 thru 0x031ea4 (0xbf3 bytes) 000879 1790728449.689981 read 0x016431 thru 0x016ca1 (0x871 bytes) 000880 1790728449.692055 mapwrite 0x011b02 thru 0x01b175 (0x9674 bytes) 000882 1790728449.763333 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33913): Mirror 7: 0xd180763a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33913): Mirror 7: 0xd180763a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33913: skipped 000883 1790728450.206316 read 0x018bb3 thru 0x0239f0 (0xae3e bytes) 000884 1790728450.226067 mapread 0x018056 thru 0x02683e (0xe7e9 bytes) 000885 1790728450.244312 punch from 0x00ffc0 to 0x014fd9 (0x501a bytes) 000887 1790728450.253033 punch from 0x02cfd5 to 0x0327f0 (0x581c bytes) 000888 1790728450.258876 read 0x01fc8e thru 0x02dc7a (0xdfed bytes) 000889 1790728450.281022 punch from 0x0236ec to 0x02bbea (0x84ff bytes) 000892 1790728450.287979 write 0x035c08 thru 0x039272 (0x366b bytes) 000894 1790728450.290586 punch from 0x00c2f4 to 0x0161ea (0x9ef7 bytes) 000895 1790728450.310576 mapread 0x005695 thru 0x012045 (0xc9b1 bytes) 000896 1790728450.354266 mapwrite 0x02dac5 thru 0x02f451 (0x198d bytes) 000899 1790728450.411456 read 0x01e178 thru 0x020e8f (0x2d18 bytes) 000901 1790728450.412515 write 0x03c902 thru 0x03ffff (0x36fe bytes) 000902 1790728450.415307 read 0x020b0f thru 0x02e18b (0xd67d bytes) 000903 1790728450.441279 MIRROR_RESYNC 2 mirrors 000905 1790728451.115030 punch from 0x03831b to 0x03c062 (0x3d48 bytes) 000906 1790728451.189349 trunc from 0x040000 to 0x02245d (0x1dba2 bytes) 000907 1790728451.234312 mapread 0x00081c thru 0x006b11 (0x62f6 bytes) 000908 1790728451.259755 trunc from 0x02245e to 0x00b93e (0x16b1f bytes) 000909 1790728451.280687 write 0x0207de thru 0x025218 (0x4a3b bytes) 000910 1790728451.281959 mapwrite 0x009ce7 thru 0x00d6e8 (0x3a02 bytes) 000911 1790728451.342410 write 0x01a14f thru 0x020509 (0x63bb bytes) 000912 1790728451.348074 read 0x020de0 thru 0x024480 (0x36a1 bytes) 000913 1790728451.348648 read 0x01a4eb thru 0x01d0bb (0x2bd1 bytes) 000914 1790728451.349199 trunc from 0x025219 to 0x02dc56 (0x8a3e bytes) 000916 1790728451.369354 write 0x03bceb thru 0x03ffff (0x4315 bytes) 000917 1790728451.376162 MIRROR_RESYNC 2 mirrors 000918 1790728452.238607 mapread 0x02a7ba thru 0x030064 (0x58ab bytes) 000919 1790728452.260138 mapread 0x0042b7 thru 0x0089d7 (0x4721 bytes) 000920 1790728452.272171 read 0x01c943 thru 0x01ff57 (0x3615 bytes) 000922 1790728452.280529 trunc from 0x040000 to 0x0359ee (0xa611 bytes) 000923 1790728452.373748 mapwrite 0x035983 thru 0x03fac2 (0xa140 bytes) 000924 1790728452.456746 write 0x03a4b8 thru 0x03ffff (0x5b48 bytes) 000925 1790728452.458940 punch from 0x03a5b4 to 0x03cb86 (0x25d3 bytes) 000927 1790728452.480734 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xdb1489a1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x19de3777 Mirror 8: 0x19de3777 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000928 1790728453.241302 write 0x03da8e thru 0x03ffff (0x2572 bytes) 000930 1790728453.354624 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xdb1489a1 (stale) Mirror 8: 0x19de3777 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa74f99f3 Mirror 9: 0xa74f99f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000931 1790728454.403715 write 0x019d39 thru 0x01f57a (0x5842 bytes) 000934 1790728454.535601 mapwrite 0x02b964 thru 0x037556 (0xbbf3 bytes) 000935 1790728454.692538 mapread 0x03321f thru 0x035dd9 (0x2bbb bytes) 000938 1790728454.697507 trunc from 0x040000 to 0x000bec (0x3f413 bytes) 000940 1790728454.725688 read 0x0007a9 thru 0x000bec (0x444 bytes) 000941 1790728454.733627 trunc from 0x000bed to 0x02d97c (0x2cd90 bytes) 000942 1790728454.763520 mapread 0x02d2bd thru 0x02d97c (0x6c0 bytes) 000943 1790728454.776721 write 0x000769 thru 0x00e5e4 (0xde7c bytes) 000944 1790728454.786606 write 0x012220 thru 0x014912 (0x26f3 bytes) 000946 1790728454.797422 punch from 0x011b14 to 0x01c2b2 (0xa79f bytes) 000947 1790728454.819813 MIRROR_RESYNC 4 mirrors 000948 1790728456.251562 trunc from 0x02d97d to 0x010b30 (0x1ce4c bytes) 000950 1790728456.335172 trunc from 0x010b31 to 0x0268e1 (0x15db1 bytes) 000951 1790728456.357292 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x268e2): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x268e2): Mirror 7: 0xecdbb676 (stale) Mirror 8: 0xecdbb676 (stale) Mirror 9: 0xecdbb676 (stale) CRC-32 checksum value for chunk [0, 0x268e2): Mirror 1: 0xecdbb676 Mirror 10: 0xecdbb676 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x268e2: skipped 000952 1790728457.540820 punch from 0x01860c to 0x0249a8 (0xc39d bytes) 000954 1790728457.643964 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x268e2): Mirror 7: 0xecdbb676 (stale) Mirror 8: 0xecdbb676 (stale) Mirror 10: 0xecdbb676 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x268e2): Mirror 7: 0xecdbb676 (stale) Mirror 8: 0xecdbb676 (stale) Mirror 10: 0xecdbb676 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x268e2: skipped 000955 1790728458.152402 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x152:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x268e2): Mirror 8: 0xecdbb676 (stale) Mirror 10: 0xecdbb676 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x268e2): Mirror 8: 0xecdbb676 (stale) Mirror 10: 0xecdbb676 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x268e2: skipped 000956 1790728458.580245 punch from 0x004d53 to 0x0122b4 (0xd562 bytes) 000957 1790728458.590587 write 0x025932 thru 0x030cea (0xb3b9 bytes) 000958 1790728458.599463 punch from 0x00d0c0 to 0x00df75 (0xeb6 bytes) 000959 1790728458.616192 read 0x007d64 thru 0x009d5a (0x1ff7 bytes) 000960 1790728458.629628 trunc from 0x030ceb to 0x0348b1 (0x3bc7 bytes) 000961 1790728458.676973 punch from 0x01d3c1 to 0x02135a (0x3f9a bytes) 000963 1790728458.682992 mapwrite 0x03633d thru 0x03cd85 (0x6a49 bytes) 000965 1790728458.899874 write 0x00b24e thru 0x018885 (0xd638 bytes) 000966 1790728458.912197 punch from 0x030e26 to 0x0343ba (0x3595 bytes) 000967 1790728458.925490 read 0x0238d6 thru 0x032e08 (0xf533 bytes) 000969 1790728458.933605 mapwrite 0x02a665 thru 0x03737b (0xcd17 bytes) 000970 1790728458.969586 mapwrite 0x01ecc8 thru 0x0233dd (0x4716 bytes) 000971 1790728459.038800 trunc from 0x03cd86 to 0x0050fb (0x37c8a bytes) 000972 1790728459.070688 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x50fc): Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x50fc): Mirror 8: 0xf19b606d (stale) Mirror 10: 0xf19b606d (stale) CRC-32 checksum value for chunk [0, 0x50fc): Mirror 1: 0x7290bee4 Mirror 11: 0x7290bee4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x50fc: skipped 000976 1790728460.150741 trunc from 0x0050fc to 0x027b34 (0x22a39 bytes) 000977 1790728460.292460 mapwrite 0x02a5aa thru 0x030024 (0x5a7b bytes) 000978 1790728460.369577 write 0x0208c5 thru 0x025a4e (0x518a bytes) 000981 1790728460.378583 mapread 0x02f073 thru 0x030024 (0xfb2 bytes) 000982 1790728460.386770 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30025): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30025): Mirror 8: 0x5711fe1b (stale) Mirror 10: 0x5711fe1b (stale) Mirror 11: 0xc93e7ebf (stale) CRC-32 checksum value for chunk [0, 0x30025): Mirror 1: 0xc73e610b Mirror 12: 0xc73e610b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30025: skipped 000984 1790728461.613819 mapread 0x007266 thru 0x00df86 (0x6d21 bytes) 000985 1790728461.656874 punch from 0x01df6e to 0x0273ad (0x9440 bytes) 000987 1790728461.729543 mapwrite 0x02fd22 thru 0x03d6da (0xd9b9 bytes) 000988 1790728461.846476 MIRROR_RESYNC 5 mirrors 000989 1790728463.209764 punch from 0x03b7aa to 0x03d6da (0x1f31 bytes) 000990 1790728463.264810 read 0x017a71 thru 0x01a374 (0x2904 bytes) 000991 1790728463.271530 mapwrite 0x002644 thru 0x00fa9d (0xd45a bytes) 000992 1790728463.346627 punch from 0x030a4d to 0x037538 (0x6aec bytes) 000995 1790728463.353336 trunc from 0x03d6db to 0x009ce7 (0x339f3 bytes) 000996 1790728463.375941 mapread 0x003aaa thru 0x009ce7 (0x623e bytes) 000998 1790728463.377780 read 0x005db0 thru 0x009ce7 (0x3f38 bytes) 000999 1790728463.379417 mapwrite 0x01cb48 thru 0x022956 (0x5e0f bytes) 001000 1790728463.466316 punch from 0x017749 to 0x01ab30 (0x33e8 bytes) All operations completed A-OK! PASS 70a (97s) == sanity-flr test 71: check mirror extend parent fid ==== 20:34:28 (1790728468) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 20:34:32 (1790728472) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.40636 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176284 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 20:34:42 (1790728482) Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5402688 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3167616 .. failed Extending file size to 8390848 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5356448 .. Extending file size to 8877792 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8904992 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8563136 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60237 60239 60240 60242 60243 done 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) 192.168.202.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:36:07 (1790728567) Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2984992 .. Extending file size to 3293312 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6131808 .. Extending file size to 7152704 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7905120 .. Extending file size to 6857664 .. Extending file size to 8218304 .. Extending file size to 8965312 .. Extending file size to 9248416 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9143232 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61835 61836 61837 61838 61839 done 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) 192.168.202.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:37:26 (1790728646) Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 2432 2206080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210560 2432 2206080 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182765 s, 11.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.428499 s, 4.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd0f57261 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:37:47 (1790728667) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102706 s, 20.4 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 20:37:54 (1790728674) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa9:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x148:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x148:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x149:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:38:01 (1790728681) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20446 s, 10.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:38:09 (1790728689) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176591 s, 11.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14b:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:38:17 (1790728697) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.327677 s, 6.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14d:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:38:50 (1790728730) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg237-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122907 s, 8.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d2:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:39:45 (1790728785) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg237-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216884 s, 9.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:40:42 (1790728842) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg237-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg237-server: Pool lustre.test_204f created Adding targets to pool oleg237-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg237-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.412963 s, 5.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg237-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg237-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg237-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:42:04 (1790728924) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:42:12 (1790728932) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:42:21 (1790728941) oleg237-server: Pool lustre.test_206 created Adding targets to pool oleg237-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg237-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg237-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg237-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg237-server: Pool lustre.test_206 destroyed PASS 206 (32s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:42:53 (1790728973) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944683 s, 11.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836591 s, 12.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd0f76467 Mirror 3: 0xd0f76467 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:43:06 (1790728986) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:43:10 (1790728990) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:43:14 (1790728994) Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0673837 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1850000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1700000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1700000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1700000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1e00000 Read this (4 bytes): BBBB 192.168.202.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg237-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:43:53 (1790729033) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0417926 s, 196 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:44:02 (1790729042) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423603 s, 24.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xca lmm_fid: [0x200000405:0xca:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xca lmm_fid: [0x200000405:0xca:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611338 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 20:44:04.000000000 -0400 Modify: 2026-09-29 20:44:04.000000000 -0400 Change: 2026-09-29 20:44:06.000000000 -0400 Birth: 2026-09-29 20:44:05.000000000 -0400 PASS 210a (13s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:44:15 (1790729055) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:44:19 (1790729059) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.671471 s, 61.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.111603 s, 36.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 29 20:44 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xcc:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 29 20:44 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:44:30 (1790729070) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:44:39 (1790729079) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1820 sec =========== 20:44:49 (1790729089) === sanity-flr: start cleanup 20:44:51 (1790729091) === === sanity-flr: finish cleanup 20:45:01 (1790729101) ===