-----============= acceptance-small: sanity-flr ============----- Fri Oct 2 01:50:26 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg209-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:50:47 (1790920247) === oleg209-client.virtnet: executing check_config_client /mnt/lustre oleg209-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg209-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ca750f3e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ca750f3e800.idle_timeout=debug disable quota as required oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:51:25 (1790920285) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7633] == sanity-flr test 0a: lfs mirror create with -N option == 01:51:28 (1790920288) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:51:50 (1790920310) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:51:55 (1790920315) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:51:59 (1790920319) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:52:29 (1790920349) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:52:33 (1790920353) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:52:37 (1790920357) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8ca750f3e800.stats=clear osc.lustre-OST0001-osc-ffff8ca750f3e800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:52:47 (1790920367) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:53:00 (1790920380) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:53:11 (1790920391) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:53:15 (1790920395) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (13s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:53:27 (1790920407) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (13s) == sanity-flr test 2: create components from existing files ========================================================== 01:53:41 (1790920421) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:53:50 (1790920430) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:53:55 (1790920435) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:54:04 (1790920444) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:54:16 (1790920456) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:54:22 (1790920462) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.31733 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224681 s, 4.7 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:54:34 (1790920474) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.505832 s, 39.4 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:54:44 (1790920484) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:54:56 (1790920496) Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ca750f3e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ca750f3e800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ca750f3e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ca750f3e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (83s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:56:19 (1790920579) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.832171 s, 13.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ca750f3e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ca750f3e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (54s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:57:13 (1790920633) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ca750f3e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ca750f3e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ca750f3e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ca750f3e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:58:39 (1790920719) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.28632 s, 9.9 MB/s -rw-r--r-- 1 root root 52428800 Oct 2 01:58 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.154s user 0m0.002s sys 0m0.201s Stopping /mnt/lustre-ost1 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8ca750f3e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8ca750f3e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.349s user 0m0.002s sys 0m0.171s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg209-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg209-server oleg209-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8ca750f3e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ca750f3e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.202s user 0m0.001s sys 0m0.178s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg209-server: oleg209-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg209-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ca750f3e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8ca750f3e800.ost_server_uuid in FULL state after 0 sec PASS 33b (108s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:00:28 (1790920828) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:00:33 (1790920833) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:00:38 (1790920838) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 02:00:42 (1790920842) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 02:00:51 (1790920851) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.873858 s, 14.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:01:07 (1790920867) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498951 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689121 s, 15.2 MB/s PASS 36b (11s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:01:18 (1790920878) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:01:27 (1790920887) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113502 s, 902 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00281276 s, 364 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000792157 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000889036 s, 1.2 MB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 02:01:40 (1790920900) create test files with size 15728965 .. 8c2e1822d23cfa1fd90576c3c0ccdfc6 - ca9459ada1c4abde3dd6301d418573db - efe6140ced193332c0aa790b34797c96 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 02:02:09 (1790920929) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:02:27 (1790920947) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:02:34 (1790920954) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880855 s, 11.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566545 s, 18.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 02:02:46 (1790920966) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28291 s, 7.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.456625 s, 9.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229522 s, 9.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 02:03:06 (1790920986) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 02:03:09 (1790920989) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:03:13 (1790920993) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472464 s, 22.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 02:03:23 (1790921003) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:03:27 (1790921007) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:03:38 (1790921018) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.281655 s, 37.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:03:50 (1790921030) llite.lustre-ffff8ca752898800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.361768 s, 29.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8ca7441c8800.xattr_cache=1 PASS 44e (19s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 02:04:09 (1790921049) error lfs split: create victim file failed: File exists PASS 44f (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:04:29 (1790921069) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (48s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:05:17 (1790921117) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (48s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:06:05 (1790921165) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 02:06:09 (1790921169) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.297984 s, 10.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.159067 s, 19.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x31baf9c4 (nosync,stale) Mirror 2: 0x31baf9c4 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 79a1e98986df319f75c222bdcb519359 - mirror 1 checksum: 925fe70b56315207136f1c7586e445a9 - mirror 2 checksum: 79a1e98986df319f75c222bdcb519359 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-02T02:06:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x105:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-02T02:06:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x106:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 79a1e98986df319f75c222bdcb519359 - mirror 1 checksum: 925fe70b56315207136f1c7586e445a9 - mirror 2 checksum: 925fe70b56315207136f1c7586e445a9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-02T02:06:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x105:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-02T02:06:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x106:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:06:23 (1790921183) Failing mds1 on oleg209-server Stopping /mnt/lustre-mds1 (opts:) on oleg209-server 02:06:30 (1790921190) shut down facet: mds1 facet_host: oleg209-server facet_failover_host: oleg209-server Failover mds1 to oleg209-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 02:07:03 (1790921223) targets are mounted 02:07:03 (1790921223) facet_failover done oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333897 s, 31.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x69 lmm_fid: [0x200000404:0x69:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x107:0x0] } PASS 50A (56s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:07:19 (1790921239) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:07:23 (1790921243) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:07:26 (1790921246) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00725177 s, 2.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:07:34 (1790921254) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:07:38 (1790921258) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00455655 s, 899 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.138906 s, 29.5 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:07:47 (1790921267) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:08:10 (1790921290) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:08:39 (1790921319) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa7:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:09:04 (1790921344) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:09:29 (1790921369) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 02:09:37 (1790921377) mirror create pid 55034 mirror split pid 55035 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 55034 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 55035 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 02:10:45 (1790921445) Seed set to 1447 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero size read skipping zero size read skipping zero size read 000005 1790921447.919909 write 0x031ab4 thru 0x03ffff (0xe54c bytes) 000007 1790921447.924117 punch from 0x02a154 to 0x02e9c7 (0x4874 bytes) 000009 1790921447.941523 mapread 0x018c2e thru 0x024687 (0xba5a bytes) 000012 1790921447.975345 mapread 0x036007 thru 0x03ffff (0x9ff9 bytes) 000014 1790921447.977517 write 0x03a066 thru 0x03ffff (0x5f9a bytes) 000016 1790921447.980621 mapwrite 0x0217d7 thru 0x02fbde (0xe408 bytes) 000018 1790921448.055419 mapread 0x039fdf thru 0x03edfa (0x4e1c bytes) 000020 1790921448.060766 mapwrite 0x01bde5 thru 0x01c29f (0x4bb bytes) 000021 1790921448.125126 read 0x00d00f thru 0x013be6 (0x6bd8 bytes) 000022 1790921448.139858 mapread 0x0176f4 thru 0x01fe1f (0x872c bytes) 000024 1790921448.143456 mapread 0x029e41 thru 0x0313a7 (0x7567 bytes) 000026 1790921448.146339 write 0x01392b thru 0x018495 (0x4b6b bytes) 000028 1790921448.149881 punch from 0x03e907 to 0x03ffff (0x16f9 bytes) truncating to largest ever: 0x1666e 000029 1790921448.170610 trunc from 0x040000 to 0x01666d (0x29992 bytes) 000032 1790921448.194001 mapwrite 0x02d56b thru 0x038f8c (0xba22 bytes) 000033 1790921448.318926 write 0x01444f thru 0x016cb6 (0x2868 bytes) 000034 1790921448.325701 MIRROR_EXTEND to 2 mirrors 000036 1790921449.489715 read 0x031af5 thru 0x038f8c (0x7498 bytes) 000037 1790921449.510018 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) 000038 1790921449.647526 mapread 0x029537 thru 0x032ce3 (0x97ad bytes) 000040 1790921449.672031 mapread 0x00592d thru 0x00c153 (0x6827 bytes) 000041 1790921449.681095 write 0x0133ce thru 0x01d119 (0x9d4c bytes) truncating to largest ever: 0x3e76c 000042 1790921449.706772 trunc from 0x038f8d to 0x03e76b (0x57df bytes) 000043 1790921449.732828 trunc from 0x03e76c to 0x017890 (0x26edb bytes) 000045 1790921449.754871 trunc from 0x017891 to 0x00686c (0x11024 bytes) 000047 1790921449.774675 punch from 0x0047d6 to 0x005bf1 (0x141c bytes) 000048 1790921449.780039 punch from 0x002b35 to 0x00686c (0x3d38 bytes) 000049 1790921449.784988 write 0x007262 thru 0x01512e (0xdecd bytes) 000051 1790921449.789194 mapread 0x00ea10 thru 0x014f43 (0x6534 bytes) 000053 1790921449.791727 punch from 0x0034db to 0x00966b (0x6191 bytes) 000056 1790921449.807732 mapwrite 0x038ff8 thru 0x03ffff (0x7008 bytes) 000057 1790921449.882243 mapwrite 0x0375d3 thru 0x03a25c (0x2c8a bytes) 000058 1790921449.924044 mapwrite 0x02c965 thru 0x030f0b (0x45a7 bytes) 000059 1790921449.972839 punch from 0x0000b4 to 0x00a3cc (0xa319 bytes) 000061 1790921449.982287 trunc from 0x040000 to 0x02ca36 (0x135c9 bytes) 000062 1790921450.013775 MIRROR_EXTEND to 2 mirrors 000064 1790921450.726243 mapread 0x00e029 thru 0x015af7 (0x7acf bytes) 000066 1790921450.783824 MIRROR_RESYNC 2 mirrors 000067 1790921451.313776 punch from 0x009b56 to 0x017d09 (0xe1b4 bytes) 000069 1790921451.403776 mapwrite 0x035fbc thru 0x03c160 (0x61a5 bytes) 000070 1790921451.576109 mapread 0x0384a6 thru 0x03c160 (0x3cbb bytes) 000072 1790921451.579778 mapread 0x030cbb thru 0x035908 (0x4c4e bytes) 000074 1790921451.589388 trunc from 0x03c161 to 0x00c72a (0x2fa36 bytes) 000075 1790921451.609219 write 0x014db7 thru 0x0224bd (0xd707 bytes) 000076 1790921451.612141 trunc from 0x0224be to 0x03d6cf (0x1b212 bytes) 000078 1790921451.635160 trunc from 0x03d6d0 to 0x01808c (0x25643 bytes) 000079 1790921451.657830 mapread 0x00e5f2 thru 0x0132d9 (0x4ce8 bytes) 000082 1790921451.672875 read 0x007a1e thru 0x013753 (0xbd36 bytes) 000084 1790921451.679446 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1808d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1808d): Mirror 2: 0x85b4382f (stale) CRC-32 checksum value for chunk [0, 0x1808d): Mirror 1: 0x5b1cd562 Mirror 3: 0x5b1cd562 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1808d: skipped 000086 1790921452.347343 punch from 0x00e747 to 0x00fdab (0x1665 bytes) 000089 1790921452.438033 MIRROR_RESYNC 3 mirrors 000092 1790921453.475756 MIRROR_RESYNC 3 mirrors 000093 1790921453.919629 punch from 0x00f767 to 0x01808c (0x8926 bytes) 000094 1790921454.011926 write 0x009a4c thru 0x013a21 (0x9fd6 bytes) 000095 1790921454.057225 mapread 0x0018ef thru 0x002fcd (0x16df bytes) 000096 1790921454.100959 write 0x038609 thru 0x03cfb1 (0x49a9 bytes) 000097 1790921454.115331 trunc from 0x03cfb2 to 0x006cf7 (0x362ba bytes) 000098 1790921454.159363 mapread 0x003dbb thru 0x003e08 (0x4e bytes) 000099 1790921454.199423 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x6cf8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6cf8): Mirror 2: 0x813c54e9 (stale) Mirror 3: 0x813c54e9 (stale) CRC-32 checksum value for chunk [0, 0x6cf8): Mirror 1: 0x813c54e9 Mirror 4: 0x813c54e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6cf8: skipped 000101 1790921455.633475 punch from 0x003556 to 0x006cf7 (0x37a2 bytes) 000102 1790921455.785023 mapwrite 0x0294d4 thru 0x0347f3 (0xb320 bytes) 000105 1790921456.017396 write 0x001369 thru 0x00cf9e (0xbc36 bytes) 000106 1790921456.036390 trunc from 0x0347f4 to 0x001bbe (0x32c35 bytes) 000107 1790921456.054752 mapwrite 0x01fb6f thru 0x028c82 (0x9114 bytes) 000109 1790921456.147674 mapread 0x00cad5 thru 0x00e843 (0x1d6f bytes) 000110 1790921456.157934 trunc from 0x028c83 to 0x01ff3a (0x8d48 bytes) 000111 1790921456.178738 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1ff3b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ff3b): Mirror 2: 0x69fd5172 (stale) Mirror 3: 0x69fd5172 (stale) Mirror 4: 0x114f92ff (stale) CRC-32 checksum value for chunk [0, 0x1ff3b): Mirror 1: 0x46b5faea Mirror 5: 0x46b5faea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ff3b: skipped 000112 1790921457.197757 read 0x00679c thru 0x0145d7 (0xde3c bytes) 000113 1790921457.261325 write 0x027e8c thru 0x033906 (0xba7b bytes) 000114 1790921457.421442 trunc from 0x033907 to 0x00c982 (0x26f84 bytes) 000115 1790921457.466404 trunc from 0x00c983 to 0x0166a0 (0x9d1e bytes) 000119 1790921457.526095 mapwrite 0x0019ea thru 0x003933 (0x1f4a bytes) 000120 1790921457.744736 write 0x0236e7 thru 0x028935 (0x524f bytes) 000121 1790921457.779560 MIRROR_RESYNC 5 mirrors 000122 1790921459.959910 trunc from 0x028936 to 0x038c61 (0x1032c bytes) 000123 1790921460.034327 mapread 0x011303 thru 0x01843d (0x713b bytes) 000124 1790921460.060129 read 0x012085 thru 0x01ea87 (0xca03 bytes) 000125 1790921460.066185 mapread 0x020c6c thru 0x02b42d (0xa7c2 bytes) 000126 1790921460.088864 mapwrite 0x01b802 thru 0x022619 (0x6e18 bytes) 000127 1790921460.144720 trunc from 0x038c62 to 0x03374a (0x5517 bytes) 000128 1790921460.169329 punch from 0x01a4aa to 0x01be23 (0x197a bytes) 000129 1790921460.175012 write 0x036cb1 thru 0x03ba9f (0x4def bytes) 000130 1790921460.182835 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3baa0): Mirror 2: 0x7d87d109 (stale) Mirror 4: 0x7d87d109 (stale) Mirror 5: 0x7d87d109 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baa0): Mirror 2: 0x7d87d109 (stale) Mirror 4: 0x7d87d109 (stale) Mirror 5: 0x7d87d109 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baa0: skipped 000131 1790921460.755039 punch from 0x02eaa2 to 0x03ba9f (0xcffe bytes) 000132 1790921460.762117 mapread 0x029e5d thru 0x02e299 (0x443d bytes) 000133 1790921460.783050 mapread 0x002e78 thru 0x0094cd (0x6656 bytes) 000134 1790921460.801663 write 0x02293d thru 0x02b286 (0x894a bytes) 000135 1790921460.807238 mapwrite 0x00e58e thru 0x00ec15 (0x688 bytes) 000136 1790921460.856367 read 0x00a157 thru 0x00f784 (0x562e bytes) 000137 1790921460.857646 read 0x01117a thru 0x013eff (0x2d86 bytes) 000139 1790921460.888729 mapwrite 0x035d3c thru 0x03a0f5 (0x43ba bytes) 000140 1790921460.944131 mapread 0x0260f1 thru 0x0305f2 (0xa502 bytes) 000141 1790921460.946102 mapread 0x00407f thru 0x012d11 (0xec93 bytes) 000142 1790921460.953894 MIRROR_RESYNC 4 mirrors 000143 1790921462.071185 punch from 0x0238d9 to 0x02b2fe (0x7a26 bytes) 000144 1790921462.170206 mapread 0x0256b2 thru 0x034542 (0xee91 bytes) 000145 1790921462.284165 read 0x0378f6 thru 0x03ba9f (0x41aa bytes) 000146 1790921462.295177 write 0x025bf5 thru 0x02ca18 (0x6e24 bytes) 000150 1790921462.314850 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3baa0): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baa0): Mirror 2: 0x27ed1835 (stale) Mirror 4: 0x27ed1835 (stale) Mirror 5: 0x27ed1835 (stale) CRC-32 checksum value for chunk [0, 0x3baa0): Mirror 1: 0x4205b11b Mirror 6: 0x4205b11b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baa0: skipped 000152 1790921463.931300 trunc from 0x03baa0 to 0x01ed03 (0x1cd9c bytes) 000153 1790921464.127982 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1ed04): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ed04): Mirror 2: 0x5e19c8 (stale) Mirror 4: 0x5e19c8 (stale) Mirror 5: 0x5e19c8 (stale) Mirror 6: 0x5e19c8 (stale) CRC-32 checksum value for chunk [0, 0x1ed04): Mirror 1: 0x5e19c8 Mirror 7: 0x5e19c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ed04: skipped 000154 1790921465.392752 mapwrite 0x0041ad thru 0x009fc3 (0x5e17 bytes) 000155 1790921465.582821 write 0x00c535 thru 0x0195cf (0xd09b bytes) 000158 1790921465.590799 punch from 0x00d6fb to 0x01059f (0x2ea5 bytes) 000159 1790921465.610725 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1ed04): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ed04): Mirror 2: 0x5e19c8 (stale) Mirror 4: 0x5e19c8 (stale) Mirror 5: 0x5e19c8 (stale) Mirror 6: 0x5e19c8 (stale) Mirror 7: 0x5e19c8 (stale) CRC-32 checksum value for chunk [0, 0x1ed04): Mirror 1: 0x5932e7 Mirror 8: 0x5932e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ed04: skipped 000161 1790921467.066954 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ed04): Mirror 2: 0x5e19c8 (stale) Mirror 4: 0x5e19c8 (stale) Mirror 5: 0x5e19c8 (stale) Mirror 6: 0x5e19c8 (stale) Mirror 7: 0x5e19c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ed04): Mirror 2: 0x5e19c8 (stale) Mirror 4: 0x5e19c8 (stale) Mirror 5: 0x5e19c8 (stale) Mirror 6: 0x5e19c8 (stale) Mirror 7: 0x5e19c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ed04: skipped 000162 1790921467.657911 MIRROR_RESYNC 6 mirrors 000163 1790921469.436466 punch from 0x011ce0 to 0x01a3fd (0x871e bytes) 000165 1790921469.634241 punch from 0x000bb8 to 0x007f63 (0x73ac bytes) 000166 1790921469.639766 mapwrite 0x03094d thru 0x03b855 (0xaf09 bytes) 000167 1790921469.902671 read 0x01be24 thru 0x026013 (0xa1f0 bytes) 000168 1790921469.934569 read 0x0295c1 thru 0x02c946 (0x3386 bytes) 000169 1790921469.970175 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b856): Mirror 4: 0xfa9fdb44 (stale) Mirror 5: 0xfa9fdb44 (stale) Mirror 6: 0xfa9fdb44 (stale) Mirror 7: 0xfa9fdb44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b856): Mirror 4: 0xfa9fdb44 (stale) Mirror 5: 0xfa9fdb44 (stale) Mirror 6: 0xfa9fdb44 (stale) Mirror 7: 0xfa9fdb44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b856: skipped 000171 1790921470.603767 mapwrite 0x0162ff thru 0x023adf (0xd7e1 bytes) 000172 1790921470.870160 write 0x0044b1 thru 0x00639a (0x1eea bytes) 000173 1790921470.895633 trunc from 0x03b856 to 0x0316a7 (0xa1ae bytes) 000174 1790921470.959510 mapwrite 0x02d97a thru 0x03ce6d (0xf4f4 bytes) 000175 1790921471.114678 punch from 0x027c9a to 0x029a03 (0x1d6a bytes) 000176 1790921471.120734 mapwrite 0x03bf8d thru 0x03ffff (0x4073 bytes) 000180 1790921471.183655 mapwrite 0x01649d thru 0x01b0ed (0x4c51 bytes) 000181 1790921471.227725 read 0x0006ac thru 0x00823c (0x7b91 bytes) 000182 1790921471.238594 mapwrite 0x00c43e thru 0x01a846 (0xe409 bytes) 000185 1790921471.286093 punch from 0x01e2be to 0x0259d9 (0x771c bytes) 000186 1790921471.306617 trunc from 0x040000 to 0x0062fe (0x39d01 bytes) 000187 1790921471.340727 mapread 0x005b4e thru 0x0062fe (0x7b1 bytes) 000189 1790921471.342201 mapwrite 0x01da98 thru 0x02a6ce (0xcc37 bytes) 000190 1790921471.546046 mapwrite 0x00912b thru 0x00e49a (0x5370 bytes) 000191 1790921471.687090 write 0x033198 thru 0x03bb86 (0x89ef bytes) 000193 1790921471.705783 read 0x028524 thru 0x0296eb (0x11c8 bytes) 000194 1790921471.710969 mapread 0x00706f thru 0x015c52 (0xebe4 bytes) 000196 1790921471.775320 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb87): Mirror 4: 0x6eb8583f (stale) Mirror 6: 0x6eb8583f (stale) Mirror 7: 0x6eb8583f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb87): Mirror 4: 0x6eb8583f (stale) Mirror 6: 0x6eb8583f (stale) Mirror 7: 0x6eb8583f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb87: skipped 000197 1790921472.347523 trunc from 0x03bb87 to 0x03b471 (0x715 bytes) 000200 1790921472.373530 punch from 0x01a5b3 to 0x01d651 (0x309f bytes) 000202 1790921472.385803 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b472): Mirror 6: 0x8b245fcf (stale) Mirror 7: 0x8b245fcf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b472): Mirror 6: 0x8b245fcf (stale) Mirror 7: 0x8b245fcf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b472: skipped 000203 1790921472.962727 write 0x000c08 thru 0x00754e (0x6947 bytes) 000204 1790921472.979701 mapwrite 0x0348cb thru 0x03ef4b (0xa681 bytes) 000205 1790921473.125541 read 0x019d0b thru 0x01bda7 (0x209d bytes) 000207 1790921473.135696 trunc from 0x03ef4c to 0x03d7e0 (0x176b bytes) 000208 1790921473.168847 mapread 0x032588 thru 0x03abbc (0x8635 bytes) 000209 1790921473.177967 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d7e1): Mirror 6: 0x9dfe6f77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7e1): Mirror 6: 0x9dfe6f77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7e1: skipped 000210 1790921473.843782 punch from 0x013131 to 0x01aea8 (0x7d78 bytes) 000211 1790921473.859238 trunc from 0x03d7e1 to 0x00f111 (0x2e6cf bytes) 000212 1790921473.889085 trunc from 0x00f112 to 0x0112ed (0x21dc bytes) 000214 1790921473.920832 read 0x00903c thru 0x00da05 (0x49ca bytes) 000215 1790921473.930038 write 0x0102e9 thru 0x018916 (0x862e bytes) 000216 1790921473.936760 trunc from 0x018917 to 0x016ec8 (0x1a4e bytes) 000217 1790921473.969907 trunc from 0x016ec9 to 0x017881 (0x9b9 bytes) 000219 1790921473.991476 punch from 0x004011 to 0x008c80 (0x4c70 bytes) 000220 1790921474.008068 read 0x011a99 thru 0x013792 (0x1cfa bytes) 000222 1790921474.010054 read 0x0152b8 thru 0x017881 (0x25ca bytes) 000224 1790921474.019917 trunc from 0x017882 to 0x008eac (0xe9d5 bytes) 000225 1790921474.047590 MIRROR_RESYNC 2 mirrors 000226 1790921475.177044 punch from 0x000cac to 0x00315a (0x24af bytes) 000227 1790921475.243780 read 0x005feb thru 0x008eac (0x2ec2 bytes) 000228 1790921475.250383 mapread 0x003af6 thru 0x007700 (0x3c0b bytes) 000229 1790921475.266700 write 0x02d8c9 thru 0x03614e (0x8886 bytes) 000231 1790921475.272503 read 0x01cb0b thru 0x01e61e (0x1b14 bytes) 000232 1790921475.303048 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3614f): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3614f): Mirror 6: 0x3f5b3879 (stale) CRC-32 checksum value for chunk [0, 0x3614f): Mirror 1: 0x8cb08e2d Mirror 7: 0x8cb08e2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3614f: skipped 000237 1790921477.218725 punch from 0x015750 to 0x019a3e (0x42ef bytes) 000238 1790921477.343192 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3614f): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3614f): Mirror 6: 0x3f5b3879 (stale) Mirror 7: 0x8cb08e2d (stale) CRC-32 checksum value for chunk [0, 0x3614f): Mirror 1: 0x8cb08e2d Mirror 8: 0x8cb08e2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3614f: skipped 000239 1790921478.369107 punch from 0x02fa7c to 0x03614e (0x66d3 bytes) 000240 1790921478.503438 punch from 0x035aa6 to 0x03614e (0x6a9 bytes) 000241 1790921478.512555 mapwrite 0x03e652 thru 0x03ffff (0x19ae bytes) 000242 1790921478.632180 read 0x03131a thru 0x03f9f8 (0xe6df bytes) 000243 1790921478.651987 trunc from 0x040000 to 0x00895b (0x376a4 bytes) 000247 1790921478.693052 punch from 0x00665e to 0x00895b (0x22fe bytes) 000248 1790921478.706138 trunc from 0x00895c to 0x01651a (0xdbbf bytes) 000249 1790921478.721575 mapread 0x0032ec thru 0x00beb1 (0x8bc6 bytes) 000251 1790921478.738133 mapwrite 0x0277c5 thru 0x028dd9 (0x1615 bytes) 000252 1790921478.925126 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x28dda): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28dda): Mirror 6: 0x57f1f301 (stale) Mirror 7: 0xf5e32fb3 (stale) Mirror 8: 0xf5e32fb3 (stale) CRC-32 checksum value for chunk [0, 0x28dda): Mirror 1: 0xdb73e26e Mirror 9: 0xdb73e26e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28dda: skipped 000253 1790921480.177546 MIRROR_RESYNC 5 mirrors 000254 1790921481.367084 punch from 0x000c58 to 0x009104 (0x84ad bytes) 000255 1790921481.434558 mapread 0x00c935 thru 0x01bb30 (0xf1fc bytes) 000256 1790921481.463000 read 0x00373c thru 0x01288a (0xf14f bytes) 000257 1790921481.470112 read 0x0206ed thru 0x027fa6 (0x78ba bytes) 000258 1790921481.485116 punch from 0x010bc5 to 0x012256 (0x1692 bytes) 000259 1790921481.489823 mapread 0x0024c8 thru 0x011ea8 (0xf9e1 bytes) 000260 1790921481.500864 write 0x0029e5 thru 0x002a1c (0x38 bytes) 000263 1790921481.503948 mapread 0x01a90f thru 0x027c53 (0xd345 bytes) 000265 1790921481.507127 punch from 0x018f69 to 0x0237bc (0xa854 bytes) 000266 1790921481.515989 mapwrite 0x037821 thru 0x03ffff (0x87df bytes) 000267 1790921481.607471 trunc from 0x040000 to 0x01e932 (0x216cd bytes) 000268 1790921481.627945 mapread 0x01c3cd thru 0x01e62b (0x225f bytes) 000270 1790921481.643899 read 0x013f60 thru 0x01cc44 (0x8ce5 bytes) 000272 1790921481.654123 mapread 0x001073 thru 0x0078bb (0x6849 bytes) 000273 1790921481.655638 punch from 0x002dcf to 0x006cef (0x3f21 bytes) 000275 1790921481.661400 trunc from 0x01e933 to 0x0390cf (0x1a79d bytes) 000276 1790921481.677507 write 0x01f9af thru 0x02f314 (0xf966 bytes) 000277 1790921481.692375 mapread 0x010026 thru 0x011488 (0x1463 bytes) 000278 1790921481.693378 trunc from 0x0390d0 to 0x023905 (0x157ca bytes) 000279 1790921481.718808 read 0x00fce8 thru 0x011a36 (0x1d4f bytes) 000280 1790921481.719658 punch from 0x007301 to 0x009cfa (0x29fa bytes) 000282 1790921481.732507 write 0x016374 thru 0x024948 (0xe5d5 bytes) 000283 1790921481.734830 write 0x00eafa thru 0x01e21b (0xf722 bytes) 000285 1790921481.737242 trunc from 0x024949 to 0x01a4ae (0xa49a bytes) 000286 1790921481.760078 mapread 0x007ffe thru 0x00a03c (0x203f bytes) 000287 1790921481.771932 mapread 0x0196cb thru 0x01a4ae (0xde4 bytes) 000288 1790921481.773961 mapwrite 0x03c833 thru 0x03ffff (0x37cd bytes) 000289 1790921481.871949 mapread 0x0294d4 thru 0x036ae4 (0xd611 bytes) 000290 1790921481.874588 read 0x012e86 thru 0x019ebb (0x7036 bytes) 000291 1790921481.875994 mapread 0x0235ea thru 0x0280b9 (0x4ad0 bytes) 000292 1790921481.878422 mapwrite 0x030059 thru 0x033e5f (0x3e07 bytes) 000293 1790921481.936744 punch from 0x036b46 to 0x03cf99 (0x6454 bytes) 000294 1790921481.949171 MIRROR_RESYNC 5 mirrors 000295 1790921483.239895 MIRROR_EXTEND to 6 mirrors 000296 1790921484.339773 punch from 0x0031a0 to 0x011b2f (0xe990 bytes) 000297 1790921484.448642 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xd2c75fec (stale) Mirror 7: 0xd2c75fec (stale) Mirror 8: 0xd2c75fec (stale) Mirror 9: 0xd2c75fec (stale) Mirror 10: 0xd2c75fec (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x15890ac8 Mirror 11: 0x15890ac8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000298 1790921485.927300 trunc from 0x040000 to 0x014867 (0x2b798 bytes) 000299 1790921486.049703 mapwrite 0x0282e9 thru 0x0326ae (0xa3c6 bytes) 000300 1790921486.201725 write 0x002783 thru 0x00c8f9 (0xa177 bytes) 000302 1790921486.232410 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326af): Mirror 6: 0x4c320666 (stale) Mirror 8: 0x4c320666 (stale) Mirror 9: 0x4c320666 (stale) Mirror 10: 0x4c320666 (stale) Mirror 11: 0x7422b630 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326af): Mirror 6: 0x4c320666 (stale) Mirror 8: 0x4c320666 (stale) Mirror 9: 0x4c320666 (stale) Mirror 10: 0x4c320666 (stale) Mirror 11: 0x7422b630 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326af: skipped 000305 1790921486.823879 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326af): Mirror 8: 0x4c320666 (stale) Mirror 9: 0x4c320666 (stale) Mirror 10: 0x4c320666 (stale) Mirror 11: 0x7422b630 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326af): Mirror 8: 0x4c320666 (stale) Mirror 9: 0x4c320666 (stale) Mirror 10: 0x4c320666 (stale) Mirror 11: 0x7422b630 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326af: skipped 000307 1790921487.290051 write 0x029d7d thru 0x0358c6 (0xbb4a bytes) 000308 1790921487.299380 mapread 0x030ad1 thru 0x032edd (0x240d bytes) 000310 1790921487.301024 read 0x009493 thru 0x012545 (0x90b3 bytes) truncating to largest ever: 0x3fe96 000312 1790921487.321446 trunc from 0x0358c7 to 0x03fe95 (0xa5cf bytes) 000313 1790921487.341672 punch from 0x012ee1 to 0x0214f6 (0xe616 bytes) 000316 1790921487.355979 read 0x0197c9 thru 0x01e28d (0x4ac5 bytes) 000318 1790921487.364969 read 0x020c09 thru 0x0246ee (0x3ae6 bytes) 000319 1790921487.365804 read 0x00a605 thru 0x01148d (0x6e89 bytes) 000323 1790921487.367738 mapread 0x0377a8 thru 0x03fe95 (0x86ee bytes) 000324 1790921487.369698 punch from 0x00dcea to 0x010ec0 (0x31d7 bytes) 000325 1790921487.374415 punch from 0x002d93 to 0x009ea4 (0x7112 bytes) 000326 1790921487.379030 write 0x019b98 thru 0x02230e (0x8777 bytes) 000327 1790921487.380731 punch from 0x02edf0 to 0x038345 (0x9556 bytes) 000330 1790921487.402832 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3fe96): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe96): Mirror 8: 0xe66f456e (stale) Mirror 9: 0xe66f456e (stale) Mirror 10: 0xe66f456e (stale) Mirror 11: 0x81bf7651 (stale) CRC-32 checksum value for chunk [0, 0x3fe96): Mirror 1: 0xc0e4afbc Mirror 12: 0xc0e4afbc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe96: skipped 000331 1790921488.282018 trunc from 0x03fe96 to 0x03728e (0x8c07 bytes) 000333 1790921488.426613 MIRROR_RESYNC 6 mirrors 000334 1790921489.647580 trunc from 0x03728f to 0x0106d1 (0x26bbd bytes) 000335 1790921489.717373 read 0x00e45d thru 0x0106d1 (0x2275 bytes) 000339 1790921489.725123 write 0x016c81 thru 0x023ad0 (0xce50 bytes) 000340 1790921489.728751 mapwrite 0x01f6b9 thru 0x02c161 (0xcaa9 bytes) 000341 1790921489.831621 mapwrite 0x033629 thru 0x038306 (0x4cde bytes) 000342 1790921489.903797 punch from 0x0259fd to 0x03097b (0xaf7f bytes) 000343 1790921489.910958 punch from 0x01df00 to 0x027ae2 (0x9be3 bytes) 000344 1790921489.915774 write 0x0215ca thru 0x02d44b (0xbe82 bytes) 000345 1790921489.926533 read 0x021e92 thru 0x0304fa (0xe669 bytes) 000346 1790921489.940597 trunc from 0x038307 to 0x00b196 (0x2d170 bytes) 000347 1790921489.963595 mapread 0x009e26 thru 0x00aab2 (0xc8d bytes) 000348 1790921489.965051 punch from 0x003952 to 0x00b196 (0x7845 bytes) 000349 1790921489.969697 mapwrite 0x019bda thru 0x028f5a (0xf381 bytes) 000350 1790921490.139523 mapwrite 0x00db38 thru 0x012578 (0x4a41 bytes) 000352 1790921490.264582 read 0x01d16f thru 0x025bc8 (0x8a5a bytes) 000354 1790921490.267026 write 0x0036ef thru 0x00e240 (0xab52 bytes) 000355 1790921490.286841 trunc from 0x028f5b to 0x0213db (0x7b7f bytes) 000356 1790921490.307646 read 0x0066ad thru 0x010839 (0xa18d bytes) 000358 1790921490.308838 write 0x01414b thru 0x01999f (0x5855 bytes) 000359 1790921490.309951 mapread 0x01d59f thru 0x0213db (0x3e3d bytes) 000360 1790921490.312871 write 0x039c7e thru 0x03b2cf (0x1652 bytes) 000361 1790921490.315107 trunc from 0x03b2d0 to 0x00a29a (0x31035 bytes) 000363 1790921490.346790 trunc from 0x00a29b to 0x03286b (0x285d1 bytes) 000364 1790921490.393565 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3286c): Mirror 8: 0x8cad658c (stale) Mirror 9: 0x8cad658c (stale) Mirror 10: 0x8cad658c (stale) Mirror 11: 0x8cad658c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3286c): Mirror 8: 0x8cad658c (stale) Mirror 9: 0x8cad658c (stale) Mirror 10: 0x8cad658c (stale) Mirror 11: 0x8cad658c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3286c: skipped 000367 1790921491.195739 punch from 0x02a3b6 to 0x03286b (0x84b6 bytes) 000368 1790921491.203852 trunc from 0x03286c to 0x02aa4a (0x7e21 bytes) 000370 1790921491.265730 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2aa4b): Mirror 8: 0xc1c57f11 (stale) Mirror 9: 0xc1c57f11 (stale) Mirror 10: 0xc1c57f11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa4b): Mirror 8: 0xc1c57f11 (stale) Mirror 9: 0xc1c57f11 (stale) Mirror 10: 0xc1c57f11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa4b: skipped 000371 1790921491.683984 trunc from 0x02aa4b to 0x0116b4 (0x19396 bytes) 000372 1790921491.700804 mapread 0x00aac8 thru 0x0116b4 (0x6bed bytes) 000374 1790921491.724373 punch from 0x005020 to 0x005e26 (0xe07 bytes) 000376 1790921491.732566 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x116b5): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x116b5): Mirror 8: 0xebd84ed3 (stale) Mirror 9: 0xebd84ed3 (stale) Mirror 10: 0xebd84ed3 (stale) CRC-32 checksum value for chunk [0, 0x116b5): Mirror 1: 0xc7526a18 Mirror 11: 0xc7526a18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x116b5: skipped 000377 1790921492.828531 write 0x022157 thru 0x02af3c (0x8de6 bytes) 000378 1790921492.942586 mapwrite 0x00093f thru 0x0087f4 (0x7eb6 bytes) 000379 1790921493.019796 write 0x0225d0 thru 0x0303fa (0xde2b bytes) 000381 1790921493.023562 mapread 0x0047a1 thru 0x0112a9 (0xcb09 bytes) 000384 1790921493.040324 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x303fb): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x303fb): Mirror 8: 0x9534f5db (stale) Mirror 9: 0x9534f5db (stale) Mirror 10: 0x9534f5db (stale) Mirror 11: 0xf5fbbc4b (stale) CRC-32 checksum value for chunk [0, 0x303fb): Mirror 1: 0x1635798a Mirror 12: 0x1635798a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x303fb: skipped 000385 1790921493.879490 trunc from 0x0303fb to 0x007dd4 (0x28626 bytes) 000386 1790921494.020574 trunc from 0x007dd5 to 0x03fdc3 (0x37fef bytes) 000388 1790921494.063172 mapwrite 0x0099ad thru 0x016390 (0xc9e4 bytes) 000389 1790921494.224569 mapwrite 0x00ed44 thru 0x01abd3 (0xbe90 bytes) 000390 1790921494.301891 mapwrite 0x03928b thru 0x03d688 (0x43fe bytes) 000391 1790921494.403821 write 0x036260 thru 0x039755 (0x34f6 bytes) 000393 1790921494.412007 mapwrite 0x00351b thru 0x00dd5e (0xa844 bytes) 000394 1790921494.518743 punch from 0x002c2a to 0x008069 (0x5440 bytes) 000397 1790921494.545710 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fdc4): Mirror 8: 0xd87b6df6 (stale) Mirror 9: 0xd87b6df6 (stale) Mirror 10: 0xd87b6df6 (stale) Mirror 11: 0xab098903 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fdc4): Mirror 8: 0xd87b6df6 (stale) Mirror 9: 0xd87b6df6 (stale) Mirror 10: 0xd87b6df6 (stale) Mirror 11: 0xab098903 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fdc4: skipped 000399 1790921495.124329 write 0x01bc53 thru 0x0240f3 (0x84a1 bytes) 000400 1790921495.140352 mapread 0x01f006 thru 0x022202 (0x31fd bytes) 000402 1790921495.142815 trunc from 0x03fdc4 to 0x02ce79 (0x12f4a bytes) 000403 1790921495.163597 mapwrite 0x013d1d thru 0x0191c1 (0x54a5 bytes) 000404 1790921495.256330 read 0x0290dc thru 0x02ce79 (0x3d9e bytes) 000406 1790921495.271674 mapread 0x0206c0 thru 0x028943 (0x8284 bytes) 000407 1790921495.280416 punch from 0x0107f3 to 0x016581 (0x5d8f bytes) 000408 1790921495.291554 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ce7a): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce7a): Mirror 8: 0x6ece4799 (stale) Mirror 9: 0x6ece4799 (stale) Mirror 10: 0x6ece4799 (stale) Mirror 11: 0x749fe396 (stale) CRC-32 checksum value for chunk [0, 0x2ce7a): Mirror 1: 0x19ae015e Mirror 12: 0x19ae015e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce7a: skipped 000409 1790921496.387240 mapread 0x02390f thru 0x02ce79 (0x956b bytes) 000410 1790921496.415394 mapwrite 0x00fd01 thru 0x01cede (0xd1de bytes) 000412 1790921496.614016 MIRROR_RESYNC 6 mirrors 000413 1790921498.190387 MIRROR_EXTEND to 7 mirrors 000415 1790921498.934677 trunc from 0x02ce7a to 0x01ae22 (0x12057 bytes) 000416 1790921499.117659 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1ae23): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ae23): Mirror 8: 0x5395204c (stale) Mirror 9: 0x5395204c (stale) Mirror 10: 0x5395204c (stale) Mirror 11: 0x5395204c (stale) Mirror 12: 0x5395204c (stale) Mirror 13: 0x5395204c (stale) CRC-32 checksum value for chunk [0, 0x1ae23): Mirror 1: 0x5395204c Mirror 14: 0x5395204c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ae23: skipped 000417 1790921500.444664 read 0x005579 thru 0x00cffb (0x7a83 bytes) 000418 1790921500.466685 punch from 0x018c88 to 0x01ae22 (0x219b bytes) 000421 1790921500.561799 punch from 0x00928b to 0x00d309 (0x407f bytes) 000422 1790921500.567175 write 0x0339a2 thru 0x039b25 (0x6184 bytes) 000424 1790921500.569627 write 0x02c687 thru 0x0378b1 (0xb22b bytes) 000425 1790921500.576966 write 0x01baab thru 0x020b03 (0x5059 bytes) 000426 1790921500.586606 mapread 0x02960a thru 0x034979 (0xb370 bytes) 000427 1790921500.603654 punch from 0x033206 to 0x039b25 (0x6920 bytes) 000429 1790921500.619064 punch from 0x031ffc to 0x039b25 (0x7b2a bytes) 000430 1790921500.624516 read 0x004450 thru 0x00d4de (0x908f bytes) 000432 1790921500.634743 write 0x010f73 thru 0x012c31 (0x1cbf bytes) 000433 1790921500.642552 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39b26): Mirror 9: 0x4e0e2e1e (stale) Mirror 10: 0x4e0e2e1e (stale) Mirror 11: 0x4e0e2e1e (stale) Mirror 12: 0x4e0e2e1e (stale) Mirror 13: 0x4e0e2e1e (stale) Mirror 14: 0x8558cb18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b26): Mirror 9: 0x4e0e2e1e (stale) Mirror 10: 0x4e0e2e1e (stale) Mirror 11: 0x4e0e2e1e (stale) Mirror 12: 0x4e0e2e1e (stale) Mirror 13: 0x4e0e2e1e (stale) Mirror 14: 0x8558cb18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b26: skipped 000434 1790921501.239834 write 0x00661b thru 0x00f557 (0x8f3d bytes) 000436 1790921501.262451 write 0x016f25 thru 0x0219d3 (0xaaaf bytes) 000437 1790921501.293515 write 0x02f517 thru 0x032577 (0x3061 bytes) 000438 1790921501.315709 MIRROR_RESYNC 7 mirrors 000439 1790921502.796550 trunc from 0x039b26 to 0x03e013 (0x44ee bytes) 000440 1790921502.888593 mapread 0x00a6c2 thru 0x013399 (0x8cd8 bytes) 000441 1790921502.916756 mapwrite 0x021337 thru 0x031237 (0xff01 bytes) 000442 1790921503.026930 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e014): Mirror 9: 0xe712a0b5 (stale) Mirror 10: 0xe712a0b5 (stale) Mirror 11: 0xe712a0b5 (stale) Mirror 12: 0xe712a0b5 (stale) Mirror 14: 0xe712a0b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e014): Mirror 9: 0xe712a0b5 (stale) Mirror 10: 0xe712a0b5 (stale) Mirror 11: 0xe712a0b5 (stale) Mirror 12: 0xe712a0b5 (stale) Mirror 14: 0xe712a0b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e014: skipped 000443 1790921503.439743 punch from 0x0155ef to 0x0188db (0x32ed bytes) 000444 1790921503.448738 read 0x011595 thru 0x01b814 (0xa280 bytes) 000446 1790921503.459292 mapread 0x02a99e thru 0x0320ed (0x7750 bytes) 000447 1790921503.483406 mapread 0x01d66d thru 0x020cc8 (0x365c bytes) 000448 1790921503.495835 write 0x0351fc thru 0x03ffff (0xae04 bytes) 000449 1790921503.502705 mapread 0x02fbc6 thru 0x039ca0 (0xa0db bytes) 000450 1790921503.503892 mapwrite 0x0221e3 thru 0x0304dd (0xe2fb bytes) 000454 1790921503.555230 read 0x034472 thru 0x03cf64 (0x8af3 bytes) 000455 1790921503.555913 mapwrite 0x00110b thru 0x002d0a (0x1c00 bytes) 000456 1790921503.601674 write 0x03dbb5 thru 0x03ffff (0x244b bytes) 000458 1790921503.603527 read 0x03e8fc thru 0x03ffff (0x1704 bytes) 000459 1790921503.607716 mapread 0x0372b2 thru 0x03ffff (0x8d4e bytes) 000461 1790921503.612178 write 0x01a370 thru 0x0223b4 (0x8045 bytes) 000462 1790921503.614486 read 0x034c27 thru 0x034fd5 (0x3af bytes) 000463 1790921503.615158 write 0x01f1db thru 0x027ade (0x8904 bytes) 000465 1790921503.617650 mapwrite 0x024989 thru 0x029cb2 (0x532a bytes) 000466 1790921503.671953 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x333c1d25 (stale) Mirror 11: 0x333c1d25 (stale) Mirror 12: 0x333c1d25 (stale) Mirror 14: 0x333c1d25 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x333c1d25 (stale) Mirror 11: 0x333c1d25 (stale) Mirror 12: 0x333c1d25 (stale) Mirror 14: 0x333c1d25 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000468 1790921504.109606 mapwrite 0x033a24 thru 0x0381b5 (0x4792 bytes) 000469 1790921504.160774 punch from 0x009436 to 0x00d05a (0x3c25 bytes) 000470 1790921504.165563 mapwrite 0x000a17 thru 0x002eb2 (0x249c bytes) 000471 1790921504.232526 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x333c1d25 (stale) Mirror 11: 0x333c1d25 (stale) Mirror 12: 0x333c1d25 (stale) Mirror 14: 0x333c1d25 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2f5a60f Mirror 15: 0x2f5a60f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000474 1790921505.088797 mapwrite 0x031309 thru 0x03ae8d (0x9b85 bytes) 000476 1790921505.269407 write 0x0393c9 thru 0x03ffff (0x6c37 bytes) 000477 1790921505.275945 MIRROR_RESYNC 6 mirrors 000480 1790921506.457901 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) 000484 1790921506.911190 MIRROR_RESYNC 5 mirrors 000485 1790921507.434503 read 0x00615e thru 0x015a3d (0xf8e0 bytes) 000486 1790921507.451829 read 0x00f6f4 thru 0x019e33 (0xa740 bytes) 000487 1790921507.468886 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) 000488 1790921507.857737 MIRROR_EXTEND to 5 mirrors 000490 1790921508.936311 MIRROR_EXTEND to 6 mirrors 000491 1790921510.179009 punch from 0x000a2d to 0x009273 (0x8847 bytes) 000492 1790921510.358758 read 0x00ee7b thru 0x01dd7f (0xef05 bytes) 000493 1790921510.380355 punch from 0x00c8b0 to 0x018964 (0xc0b5 bytes) 000496 1790921510.397715 trunc from 0x040000 to 0x0376d5 (0x892a bytes) 000497 1790921510.433825 read 0x027a04 thru 0x027a12 (0xf bytes) 000498 1790921510.449923 read 0x00d3f5 thru 0x01c523 (0xf12f bytes) 000499 1790921510.462684 MIRROR_RESYNC 6 mirrors 000500 1790921511.876528 mapwrite 0x013326 thru 0x01c3cf (0x90aa bytes) 000503 1790921512.064662 punch from 0x021bee to 0x0252b1 (0x36c4 bytes) 000504 1790921512.070497 read 0x002f8c thru 0x004bea (0x1c5f bytes) 000510 1790921512.087482 punch from 0x0153bf to 0x016df4 (0x1a36 bytes) 000511 1790921512.096487 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x376d6): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376d6): Mirror 10: 0x1e3d65a7 (stale) Mirror 11: 0x1e3d65a7 (stale) Mirror 12: 0x1e3d65a7 (stale) Mirror 13: 0x1e3d65a7 (stale) Mirror 14: 0x1e3d65a7 (stale) CRC-32 checksum value for chunk [0, 0x376d6): Mirror 1: 0xa5d7bebe Mirror 15: 0xa5d7bebe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376d6: skipped 000512 1790921513.190006 write 0x02ff3c thru 0x03d732 (0xd7f7 bytes) 000514 1790921513.279552 mapwrite 0x009499 thru 0x00b698 (0x2200 bytes) 000515 1790921513.333865 read 0x02f616 thru 0x032717 (0x3102 bytes) 000518 1790921513.338052 punch from 0x0387f8 to 0x039d47 (0x1550 bytes) 000519 1790921513.346855 MIRROR_RESYNC 7 mirrors 000521 1790921514.648424 write 0x000124 thru 0x00eabc (0xe999 bytes) 000522 1790921514.745173 write 0x02e257 thru 0x03def4 (0xfc9e bytes) 000523 1790921514.756904 mapread 0x02c92f thru 0x03040c (0x3ade bytes) 000524 1790921514.771699 mapwrite 0x03c74f thru 0x03ffff (0x38b1 bytes) 000527 1790921514.866140 write 0x02b233 thru 0x02fd3e (0x4b0c bytes) 000528 1790921514.873539 punch from 0x0166e5 to 0x019bc6 (0x34e2 bytes) 000529 1790921514.888100 punch from 0x01de6d to 0x025737 (0x78cb bytes) 000530 1790921514.897032 MIRROR_RESYNC 7 mirrors 000532 1790921516.572546 mapread 0x03b103 thru 0x03f7d3 (0x46d1 bytes) 000533 1790921516.619670 mapread 0x00595e thru 0x00d2f9 (0x799c bytes) 000534 1790921516.637980 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) 000535 1790921517.194410 mapwrite 0x020139 thru 0x02c766 (0xc62e bytes) 000536 1790921517.621351 mapwrite 0x03861c thru 0x03e81b (0x6200 bytes) 000538 1790921517.695481 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x79684bc2 (stale) Mirror 12: 0x79684bc2 (stale) Mirror 13: 0x79684bc2 (stale) Mirror 14: 0x79684bc2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x79684bc2 (stale) Mirror 12: 0x79684bc2 (stale) Mirror 13: 0x79684bc2 (stale) Mirror 14: 0x79684bc2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000539 1790921518.084825 mapwrite 0x030780 thru 0x0326ff (0x1f80 bytes) 000540 1790921518.141360 read 0x02d1f7 thru 0x033bcf (0x69d9 bytes) 000542 1790921518.142563 punch from 0x03ac80 to 0x03ffff (0x5380 bytes) 000543 1790921518.149586 trunc from 0x040000 to 0x009aa2 (0x3655d bytes) 000544 1790921518.165915 read 0x0096b2 thru 0x009aa2 (0x3f1 bytes) 000546 1790921518.180148 mapread 0x000492 thru 0x004586 (0x40f5 bytes) 000547 1790921518.185659 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9aa3): Mirror 11: 0x5bf9bce0 (stale) Mirror 12: 0x5bf9bce0 (stale) Mirror 13: 0x5bf9bce0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9aa3): Mirror 11: 0x5bf9bce0 (stale) Mirror 12: 0x5bf9bce0 (stale) Mirror 13: 0x5bf9bce0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9aa3: skipped 000549 1790921518.503003 write 0x01ad31 thru 0x023ddc (0x90ac bytes) 000550 1790921518.504849 trunc from 0x023ddd to 0x0125b7 (0x11825 bytes) 000551 1790921518.525218 punch from 0x004d38 to 0x00c960 (0x7c29 bytes) 000552 1790921518.530630 trunc from 0x0125b8 to 0x002a93 (0xfb24 bytes) 000553 1790921518.553993 write 0x012070 thru 0x0167f3 (0x4784 bytes) 000554 1790921518.556528 punch from 0x01356d to 0x013ca5 (0x739 bytes) 000555 1790921518.574241 punch from 0x00c13f to 0x010734 (0x45f6 bytes) 000556 1790921518.579472 mapread 0x00578a thru 0x005e06 (0x67d bytes) 000557 1790921518.586125 punch from 0x01396c to 0x0167f3 (0x2e88 bytes) 000559 1790921518.592452 mapread 0x00a7bf thru 0x0125c3 (0x7e05 bytes) 000560 1790921518.612048 punch from 0x009fcb to 0x0167f3 (0xc829 bytes) 000561 1790921518.617225 trunc from 0x0167f4 to 0x032ce4 (0x1c4f1 bytes) 000567 1790921518.637480 punch from 0x01e1ab to 0x01ffdf (0x1e35 bytes) 000568 1790921518.642460 write 0x02f017 thru 0x031cdd (0x2cc7 bytes) 000570 1790921518.653105 mapread 0x019673 thru 0x021d0b (0x8699 bytes) 000571 1790921518.672946 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32ce5): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ce5): Mirror 11: 0x46654491 (stale) Mirror 12: 0x46654491 (stale) Mirror 13: 0x46654491 (stale) CRC-32 checksum value for chunk [0, 0x32ce5): Mirror 1: 0x5c32980f Mirror 14: 0x5c32980f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ce5: skipped 000572 1790921519.689807 trunc from 0x032ce5 to 0x012aff (0x201e5 bytes) 000574 1790921519.846529 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12b00): Mirror 11: 0xbe4a29d7 (stale) Mirror 12: 0xbe4a29d7 (stale) Mirror 14: 0xae4ba048 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12b00): Mirror 11: 0xbe4a29d7 (stale) Mirror 12: 0xbe4a29d7 (stale) Mirror 14: 0xae4ba048 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12b00: skipped 000575 1790921520.191627 mapread 0x00cb20 thru 0x012aff (0x5fe0 bytes) 000576 1790921520.213189 punch from 0x012a5a to 0x012aff (0xa6 bytes) 000577 1790921520.221780 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x12b00): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12b00): Mirror 11: 0xbe4a29d7 (stale) Mirror 12: 0xbe4a29d7 (stale) Mirror 14: 0xae4ba048 (stale) CRC-32 checksum value for chunk [0, 0x12b00): Mirror 1: 0xae4ba048 Mirror 15: 0xae4ba048 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12b00: skipped 000579 1790921521.141583 trunc from 0x012b00 to 0x03931b (0x2681c bytes) 000581 1790921521.266747 mapread 0x028886 thru 0x02a559 (0x1cd4 bytes) 000582 1790921521.329991 trunc from 0x03931c to 0x01b998 (0x1d983 bytes) 000583 1790921521.399932 punch from 0x005bcd to 0x0125e4 (0xca18 bytes) 000584 1790921521.409429 punch from 0x008226 to 0x0103a1 (0x817c bytes) 000585 1790921521.429365 mapread 0x00b4fc thru 0x015590 (0xa095 bytes) 000586 1790921521.474673 trunc from 0x01b999 to 0x017e01 (0x3b97 bytes) 000587 1790921521.502837 mapread 0x003300 thru 0x0087b8 (0x54b9 bytes) 000589 1790921521.524151 mapwrite 0x02bf09 thru 0x02f9c6 (0x3abe bytes) 000590 1790921521.685717 mapread 0x01d2f9 thru 0x01d473 (0x17b bytes) 000591 1790921521.708710 write 0x00cbf5 thru 0x01bf30 (0xf33c bytes) 000592 1790921521.710148 punch from 0x011e14 to 0x01432c (0x2519 bytes) 000594 1790921521.769656 mapwrite 0x004996 thru 0x007865 (0x2ed0 bytes) 000595 1790921521.832612 trunc from 0x02f9c7 to 0x010e20 (0x1eba6 bytes) 000596 1790921521.889812 trunc from 0x010e21 to 0x02ebdf (0x1ddbf bytes) 000597 1790921521.922525 write 0x03389c thru 0x03d3ec (0x9b51 bytes) 000598 1790921521.926834 read 0x0333a3 thru 0x037993 (0x45f1 bytes) 000599 1790921521.928223 write 0x0004a9 thru 0x00e0f7 (0xdc4f bytes) 000600 1790921521.941857 write 0x02d791 thru 0x03650c (0x8d7c bytes) 000601 1790921521.953584 read 0x027a74 thru 0x0329c4 (0xaf51 bytes) 000605 1790921521.987002 punch from 0x01e618 to 0x02acea (0xc6d3 bytes) 000606 1790921522.034072 trunc from 0x03d3ed to 0x00945b (0x33f91 bytes) 000607 1790921522.084071 trunc from 0x00945c to 0x031efd (0x28aa2 bytes) 000609 1790921522.121382 trunc from 0x031efe to 0x00b72b (0x267d2 bytes) 000611 1790921522.160492 mapread 0x005383 thru 0x00b72b (0x63a9 bytes) 000614 1790921522.187685 punch from 0x0090cf to 0x009e24 (0xd56 bytes) 000616 1790921522.209570 mapwrite 0x00472c thru 0x0072b4 (0x2b89 bytes) 000618 1790921522.339164 mapread 0x006eaf thru 0x00b72b (0x487d bytes) 000619 1790921522.350848 punch from 0x00a1e4 to 0x00b72b (0x1548 bytes) 000620 1790921522.395683 MIRROR_RESYNC 5 mirrors 000621 1790921524.271440 mapwrite 0x003f74 thru 0x007cab (0x3d38 bytes) 000622 1790921524.533179 mapread 0x00ad79 thru 0x00b72b (0x9b3 bytes) 000623 1790921524.534761 trunc from 0x00b72c to 0x01a079 (0xe94e bytes) 000624 1790921524.577427 mapread 0x01703a thru 0x01a079 (0x3040 bytes) 000625 1790921524.591438 trunc from 0x01a07a to 0x003956 (0x16723 bytes) 000626 1790921524.656948 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3957): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3957): Mirror 11: 0xce7196a6 (stale) Mirror 12: 0xce7196a6 (stale) Mirror 14: 0xce7196a6 (stale) Mirror 15: 0xce7196a6 (stale) CRC-32 checksum value for chunk [0, 0x3957): Mirror 1: 0xce7196a6 Mirror 16: 0xce7196a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3957: skipped 000627 1790921525.833127 mapread 0x0014da thru 0x0018e6 (0x40d bytes) 000629 1790921525.856762 trunc from 0x003957 to 0x011fdd (0xe687 bytes) 000632 1790921525.934031 write 0x00c1e3 thru 0x018e2b (0xcc49 bytes) 000633 1790921525.942462 read 0x00df12 thru 0x0156ba (0x77a9 bytes) 000639 1790921525.949448 write 0x02bba6 thru 0x034c91 (0x90ec bytes) 000640 1790921525.951723 trunc from 0x034c92 to 0x005f3d (0x2ed54 bytes) 000641 1790921525.970061 mapread 0x0056d2 thru 0x005f3d (0x86c bytes) 000642 1790921525.976625 mapread 0x001768 thru 0x005f3d (0x47d6 bytes) 000644 1790921525.983769 read 0x0002ec thru 0x005f3d (0x5c52 bytes) 000645 1790921525.994109 mapwrite 0x0083bd thru 0x012254 (0x9e98 bytes) 000648 1790921526.086353 MIRROR_RESYNC 6 mirrors 000650 1790921527.630487 MIRROR_RESYNC 6 mirrors 000651 1790921528.086722 write 0x0176fa thru 0x023cd4 (0xc5db bytes) 000655 1790921528.161642 read 0x022d18 thru 0x023cd4 (0xfbd bytes) 000657 1790921528.164077 write 0x033d15 thru 0x03ffff (0xc2eb bytes) 000658 1790921528.169593 mapwrite 0x02e7e0 thru 0x03a738 (0xbf59 bytes) 000661 1790921528.251111 write 0x030b15 thru 0x03d569 (0xca55 bytes) 000662 1790921528.252772 mapread 0x02276b thru 0x027487 (0x4d1d bytes) 000663 1790921528.264990 mapwrite 0x01b0f3 thru 0x020ec7 (0x5dd5 bytes) 000664 1790921528.325498 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x2574614b (stale) Mirror 14: 0x2574614b (stale) Mirror 15: 0x2574614b (stale) Mirror 16: 0x2574614b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x2574614b (stale) Mirror 14: 0x2574614b (stale) Mirror 15: 0x2574614b (stale) Mirror 16: 0x2574614b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000665 1790921528.712641 mapread 0x02ad0e thru 0x036ab7 (0xbdaa bytes) 000666 1790921528.745209 mapwrite 0x03fdc0 thru 0x03ffff (0x240 bytes) 000667 1790921528.789220 read 0x00d873 thru 0x00da19 (0x1a7 bytes) 000668 1790921528.795001 mapread 0x03452a thru 0x03e038 (0x9b0f bytes) 000670 1790921528.797656 punch from 0x000abe to 0x007348 (0x688b bytes) 000671 1790921528.802975 mapread 0x02fad9 thru 0x03ee54 (0xf37c bytes) 000675 1790921528.810621 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x2574614b (stale) Mirror 14: 0x2574614b (stale) Mirror 15: 0x2574614b (stale) Mirror 16: 0x2574614b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5fa05c6f Mirror 17: 0x5fa05c6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000676 1790921529.660481 punch from 0x01686d to 0x0246e7 (0xde7b bytes) 000677 1790921529.723228 punch from 0x004c07 to 0x007d25 (0x311f bytes) 000678 1790921529.728722 write 0x02ca6d thru 0x032293 (0x5827 bytes) 000683 1790921529.739942 write 0x0124f8 thru 0x0188bb (0x63c4 bytes) 000684 1790921529.749453 trunc from 0x040000 to 0x002c5c (0x3d3a3 bytes) 000685 1790921529.772107 trunc from 0x002c5d to 0x026d76 (0x2411a bytes) 000688 1790921529.801696 MIRROR_RESYNC 6 mirrors 000689 1790921531.241054 trunc from 0x026d77 to 0x00c7e9 (0x1a58d bytes) 000690 1790921531.380641 trunc from 0x00c7ea to 0x01831e (0xbb35 bytes) 000691 1790921531.412454 punch from 0x005eb5 to 0x0082e7 (0x2433 bytes) 000693 1790921531.419098 punch from 0x0025db to 0x0083cc (0x5df2 bytes) 000694 1790921531.429054 read 0x017a4c thru 0x01831e (0x8d3 bytes) 000695 1790921531.443928 mapwrite 0x00d393 thru 0x010ea8 (0x3b16 bytes) 000696 1790921531.690937 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1831f): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1831f): Mirror 12: 0xa5e61ebe (stale) Mirror 14: 0xa5e61ebe (stale) Mirror 15: 0xa5e61ebe (stale) Mirror 16: 0xa5e61ebe (stale) Mirror 17: 0xa5e61ebe (stale) CRC-32 checksum value for chunk [0, 0x1831f): Mirror 1: 0x9df2136a Mirror 18: 0x9df2136a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1831f: skipped 000698 1790921532.947865 read 0x00c43b thru 0x01831e (0xbee4 bytes) 000699 1790921532.976220 trunc from 0x01831f to 0x016d23 (0x15fb bytes) 000700 1790921533.139709 trunc from 0x016d24 to 0x01a6dc (0x39b9 bytes) 000701 1790921533.221629 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1a6dd): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a6dd): Mirror 12: 0x81e49eb4 (stale) Mirror 14: 0x81e49eb4 (stale) Mirror 15: 0x81e49eb4 (stale) Mirror 16: 0x81e49eb4 (stale) Mirror 17: 0x81e49eb4 (stale) Mirror 18: 0xd6592d8a (stale) CRC-32 checksum value for chunk [0, 0x1a6dd): Mirror 1: 0xd6592d8a Mirror 19: 0xd6592d8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a6dd: skipped 000703 1790921534.982197 punch from 0x0100ea to 0x01794b (0x7862 bytes) 000705 1790921535.183126 mapwrite 0x03b640 thru 0x03ffff (0x49c0 bytes) 000706 1790921535.416155 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x385a3fdb (stale) Mirror 14: 0x385a3fdb (stale) Mirror 15: 0x385a3fdb (stale) Mirror 16: 0x385a3fdb (stale) Mirror 17: 0x385a3fdb (stale) Mirror 18: 0x8f70de8f (stale) Mirror 19: 0x8f70de8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x43175680 Mirror 20: 0x43175680 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000707 1790921536.727500 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x385a3fdb (stale) Mirror 14: 0x385a3fdb (stale) Mirror 15: 0x385a3fdb (stale) Mirror 16: 0x385a3fdb (stale) Mirror 17: 0x385a3fdb (stale) Mirror 19: 0x8f70de8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x385a3fdb (stale) Mirror 14: 0x385a3fdb (stale) Mirror 15: 0x385a3fdb (stale) Mirror 16: 0x385a3fdb (stale) Mirror 17: 0x385a3fdb (stale) Mirror 19: 0x8f70de8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x43175680 Mirror 20: 0x43175680 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000708 1790921537.951825 trunc from 0x040000 to 0x010df0 (0x2f20f bytes) 000709 1790921538.085870 read 0x009630 thru 0x010df0 (0x77c1 bytes) 000710 1790921538.096878 mapread 0x007494 thru 0x010df0 (0x995d bytes) 000711 1790921538.110535 write 0x001977 thru 0x008106 (0x6790 bytes) 000712 1790921538.121513 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10df1): Mirror 12: 0x95971fc9 (stale) Mirror 14: 0x95971fc9 (stale) Mirror 15: 0x95971fc9 (stale) Mirror 16: 0x95971fc9 (stale) Mirror 17: 0x95971fc9 (stale) Mirror 20: 0x4635cff2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10df1): Mirror 12: 0x95971fc9 (stale) Mirror 14: 0x95971fc9 (stale) Mirror 15: 0x95971fc9 (stale) Mirror 16: 0x95971fc9 (stale) Mirror 17: 0x95971fc9 (stale) Mirror 20: 0x4635cff2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10df1: skipped 000713 1790921538.664977 trunc from 0x010df1 to 0x005705 (0xb6eb bytes) 000715 1790921538.681841 mapwrite 0x025ab4 thru 0x02a6b5 (0x4c02 bytes) 000717 1790921538.801993 mapread 0x017e73 thru 0x022147 (0xa2d5 bytes) 000718 1790921538.814451 trunc from 0x02a6b6 to 0x030b1d (0x6468 bytes) 000719 1790921538.842551 punch from 0x024048 to 0x024a41 (0x9fa bytes) 000721 1790921538.853529 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30b1e): Mirror 12: 0x572b33d9 (stale) Mirror 15: 0x572b33d9 (stale) Mirror 16: 0x572b33d9 (stale) Mirror 17: 0x572b33d9 (stale) Mirror 20: 0xcf2e0ce2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b1e): Mirror 12: 0x572b33d9 (stale) Mirror 15: 0x572b33d9 (stale) Mirror 16: 0x572b33d9 (stale) Mirror 17: 0x572b33d9 (stale) Mirror 20: 0xcf2e0ce2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b1e: skipped 000723 1790921539.448893 read 0x01ce39 thru 0x021ca7 (0x4e6f bytes) 000724 1790921539.472473 MIRROR_RESYNC 6 mirrors 000725 1790921541.429909 read 0x018c48 thru 0x01d8e3 (0x4c9c bytes) 000726 1790921541.445353 trunc from 0x030b1e to 0x02e5e4 (0x2539 bytes) 000727 1790921541.531242 read 0x01dadf thru 0x02a827 (0xcd49 bytes) 000728 1790921541.549911 mapread 0x0212ea thru 0x0284eb (0x7202 bytes) 000729 1790921541.551942 read 0x007f79 thru 0x00a980 (0x2a08 bytes) 000730 1790921541.552859 mapread 0x015370 thru 0x019658 (0x42e9 bytes) 000731 1790921541.555297 trunc from 0x02e5e5 to 0x0102cb (0x1e319 bytes) 000732 1790921541.602953 trunc from 0x0102cc to 0x00cf78 (0x3353 bytes) 000733 1790921541.618954 mapwrite 0x02cf78 thru 0x02f2d1 (0x235a bytes) 000736 1790921541.681116 write 0x032713 thru 0x033416 (0xd04 bytes) 000737 1790921541.682528 mapwrite 0x030504 thru 0x0308d4 (0x3d1 bytes) 000738 1790921541.744624 mapwrite 0x01adf3 thru 0x02652e (0xb73c bytes) 000739 1790921541.886484 MIRROR_RESYNC 6 mirrors 000740 1790921543.586539 MIRROR_EXTEND to 7 mirrors 000741 1790921544.288126 read 0x002a65 thru 0x007d73 (0x530f bytes) 000743 1790921544.299081 punch from 0x02e404 to 0x033416 (0x5013 bytes) 000744 1790921544.376221 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33417): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33417): Mirror 12: 0xe4ea3592 (stale) Mirror 15: 0xe4ea3592 (stale) Mirror 16: 0xe4ea3592 (stale) Mirror 17: 0xe4ea3592 (stale) Mirror 20: 0xe4ea3592 (stale) Mirror 21: 0xe4ea3592 (stale) CRC-32 checksum value for chunk [0, 0x33417): Mirror 1: 0x663aea0f Mirror 22: 0x663aea0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33417: skipped 000745 1790921545.437015 read 0x01a85e thru 0x02926c (0xea0f bytes) 000747 1790921545.462054 mapread 0x00fa97 thru 0x01580d (0x5d77 bytes) 000749 1790921545.502245 trunc from 0x033417 to 0x014125 (0x1f2f1 bytes) 000752 1790921545.589974 mapwrite 0x01bcf3 thru 0x02391f (0x7c2d bytes) 000753 1790921545.667238 punch from 0x008f6f to 0x014eee (0xbf80 bytes) 000758 1790921545.680833 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x23920): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23920): Mirror 12: 0xb152af1 (stale) Mirror 15: 0xb152af1 (stale) Mirror 16: 0xb152af1 (stale) Mirror 17: 0xb152af1 (stale) Mirror 20: 0xb152af1 (stale) Mirror 21: 0xb152af1 (stale) Mirror 22: 0xb152af1 (stale) CRC-32 checksum value for chunk [0, 0x23920): Mirror 1: 0x52d317b1 Mirror 23: 0x52d317b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23920: skipped 000760 1790921546.713691 read 0x0111d5 thru 0x01b5be (0xa3ea bytes) 000762 1790921546.788958 trunc from 0x023920 to 0x03c100 (0x187e1 bytes) 000764 1790921546.925969 mapread 0x039cb0 thru 0x03c100 (0x2451 bytes) 000766 1790921546.955575 read 0x015130 thru 0x024f38 (0xfe09 bytes) 000767 1790921546.976866 mapread 0x038f0f thru 0x03c100 (0x31f2 bytes) 000768 1790921546.978922 trunc from 0x03c101 to 0x015b5f (0x265a1 bytes) 000769 1790921547.005175 punch from 0x01235e to 0x015b5f (0x3802 bytes) 000771 1790921547.016981 mapwrite 0x016a22 thru 0x01ddb1 (0x7390 bytes) 000772 1790921547.159328 mapread 0x007a25 thru 0x00a385 (0x2961 bytes) 000774 1790921547.162043 mapwrite 0x03783f thru 0x03ffff (0x87c1 bytes) 000779 1790921547.257219 read 0x03eea5 thru 0x03ffff (0x115b bytes) 000780 1790921547.260099 write 0x01516f thru 0x021e4f (0xcce1 bytes) 000782 1790921547.287153 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xe8aff297 (stale) Mirror 15: 0xe8aff297 (stale) Mirror 16: 0xe8aff297 (stale) Mirror 17: 0xe8aff297 (stale) Mirror 20: 0xe8aff297 (stale) Mirror 21: 0xe8aff297 (stale) Mirror 22: 0x81fb61b6 (stale) Mirror 23: 0xed260afa (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9355fa55 Mirror 24: 0x9355fa55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000783 1790921548.279756 punch from 0x033881 to 0x03ffb5 (0xc735 bytes) 000784 1790921548.382326 mapread 0x00d44c thru 0x016b61 (0x9716 bytes) 000785 1790921548.404795 mapwrite 0x02e7f7 thru 0x0333b5 (0x4bbf bytes) 000786 1790921548.488848 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xe8aff297 (stale) Mirror 15: 0xe8aff297 (stale) Mirror 16: 0xe8aff297 (stale) Mirror 17: 0xe8aff297 (stale) Mirror 20: 0xe8aff297 (stale) Mirror 21: 0xe8aff297 (stale) Mirror 22: 0x81fb61b6 (stale) Mirror 23: 0xed260afa (stale) Mirror 24: 0x9355fa55 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xca635c9b Mirror 25: 0xca635c9b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000787 1790921549.333958 trunc from 0x040000 to 0x01eda5 (0x2125a bytes) 000788 1790921549.467005 write 0x01610c thru 0x01fa31 (0x9926 bytes) 000789 1790921549.474874 mapwrite 0x03e2eb thru 0x03ffff (0x1d15 bytes) 000790 1790921549.561702 MIRROR_RESYNC 11 mirrors 000791 1790921552.341197 read 0x03169b thru 0x034beb (0x3551 bytes) 000793 1790921552.348805 mapwrite 0x036757 thru 0x03ad54 (0x45fe bytes) 000796 1790921552.679455 trunc from 0x040000 to 0x002b1d (0x3d4e2 bytes) 000798 1790921552.718486 trunc from 0x002b1e to 0x037f7d (0x35460 bytes) 000800 1790921552.768840 read 0x021bf1 thru 0x02d3c6 (0xb7d6 bytes) 000801 1790921552.792097 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37f7e): Mirror 12: 0xff1c76c9 (stale) Mirror 15: 0xff1c76c9 (stale) Mirror 16: 0xff1c76c9 (stale) Mirror 17: 0xff1c76c9 (stale) Mirror 20: 0xff1c76c9 (stale) Mirror 21: 0xff1c76c9 (stale) Mirror 22: 0xff1c76c9 (stale) Mirror 23: 0xff1c76c9 (stale) Mirror 24: 0xff1c76c9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f7e): Mirror 12: 0xff1c76c9 (stale) Mirror 15: 0xff1c76c9 (stale) Mirror 16: 0xff1c76c9 (stale) Mirror 17: 0xff1c76c9 (stale) Mirror 20: 0xff1c76c9 (stale) Mirror 21: 0xff1c76c9 (stale) Mirror 22: 0xff1c76c9 (stale) Mirror 23: 0xff1c76c9 (stale) Mirror 24: 0xff1c76c9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f7e: skipped 000802 1790921553.402403 punch from 0x0060fd to 0x013232 (0xd136 bytes) 000804 1790921553.406872 mapread 0x01d3e5 thru 0x026683 (0x929f bytes) 000808 1790921553.427185 write 0x03f048 thru 0x03ffff (0xfb8 bytes) 000809 1790921553.435691 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7e3e9864 (stale) Mirror 15: 0x7e3e9864 (stale) Mirror 16: 0x7e3e9864 (stale) Mirror 17: 0x7e3e9864 (stale) Mirror 20: 0x7e3e9864 (stale) Mirror 21: 0x7e3e9864 (stale) Mirror 22: 0x7e3e9864 (stale) Mirror 23: 0x7e3e9864 (stale) Mirror 24: 0x7e3e9864 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x45263e77 Mirror 25: 0x45263e77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000810 1790921554.875695 write 0x005b99 thru 0x00b43e (0x58a6 bytes) 000811 1790921554.988681 write 0x0105f4 thru 0x020256 (0xfc63 bytes) 000812 1790921554.999742 mapread 0x00bb47 thru 0x00c301 (0x7bb bytes) 000813 1790921555.009040 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7e3e9864 (stale) Mirror 15: 0x7e3e9864 (stale) Mirror 16: 0x7e3e9864 (stale) Mirror 17: 0x7e3e9864 (stale) Mirror 20: 0x7e3e9864 (stale) Mirror 22: 0x7e3e9864 (stale) Mirror 23: 0x7e3e9864 (stale) Mirror 24: 0x7e3e9864 (stale) Mirror 25: 0x45263e77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7e3e9864 (stale) Mirror 15: 0x7e3e9864 (stale) Mirror 16: 0x7e3e9864 (stale) Mirror 17: 0x7e3e9864 (stale) Mirror 20: 0x7e3e9864 (stale) Mirror 22: 0x7e3e9864 (stale) Mirror 23: 0x7e3e9864 (stale) Mirror 24: 0x7e3e9864 (stale) Mirror 25: 0x45263e77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000815 1790921555.553884 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7e3e9864 (stale) Mirror 15: 0x7e3e9864 (stale) Mirror 16: 0x7e3e9864 (stale) Mirror 17: 0x7e3e9864 (stale) Mirror 20: 0x7e3e9864 (stale) Mirror 22: 0x7e3e9864 (stale) Mirror 23: 0x7e3e9864 (stale) Mirror 24: 0x7e3e9864 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7e3e9864 (stale) Mirror 15: 0x7e3e9864 (stale) Mirror 16: 0x7e3e9864 (stale) Mirror 17: 0x7e3e9864 (stale) Mirror 20: 0x7e3e9864 (stale) Mirror 22: 0x7e3e9864 (stale) Mirror 23: 0x7e3e9864 (stale) Mirror 24: 0x7e3e9864 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000816 1790921556.208172 MIRROR_RESYNC 9 mirrors 000817 1790921557.831726 write 0x010ad5 thru 0x012a92 (0x1fbe bytes) 000820 1790921557.903767 mapread 0x0394da thru 0x03ffff (0x6b26 bytes) 000821 1790921557.924060 write 0x008510 thru 0x013904 (0xb3f5 bytes) 000822 1790921557.932696 punch from 0x02b511 to 0x03a2d7 (0xedc7 bytes) 000824 1790921557.944766 punch from 0x01ccd1 to 0x02246a (0x579a bytes) 000825 1790921557.952041 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xfd05edf6 (stale) Mirror 15: 0xfd05edf6 (stale) Mirror 16: 0xfd05edf6 (stale) Mirror 17: 0xfd05edf6 (stale) Mirror 20: 0xfd05edf6 (stale) Mirror 22: 0xfd05edf6 (stale) Mirror 23: 0xfd05edf6 (stale) Mirror 24: 0xfd05edf6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfad6f54c Mirror 25: 0xfad6f54c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000826 1790921559.366818 mapread 0x010475 thru 0x01d7a0 (0xd32c bytes) 000828 1790921559.415014 trunc from 0x040000 to 0x00c6a7 (0x33958 bytes) 000829 1790921559.507117 mapread 0x00c067 thru 0x00c6a7 (0x641 bytes) 000830 1790921559.517648 punch from 0x000c0c to 0x001cc1 (0x10b6 bytes) 000831 1790921559.526069 MIRROR_RESYNC 10 mirrors 000833 1790921561.446496 mapread 0x00935c thru 0x00c6a7 (0x334c bytes) 000836 1790921561.474391 mapread 0x0083ea thru 0x00a7a0 (0x23b7 bytes) 000837 1790921561.475216 write 0x0191e0 thru 0x0267a4 (0xd5c5 bytes) 000838 1790921561.538229 punch from 0x025a92 to 0x0267a4 (0xd13 bytes) 000841 1790921561.564831 mapread 0x011715 thru 0x01cb9d (0xb489 bytes) 000843 1790921561.577078 write 0x03ad59 thru 0x03ffff (0x52a7 bytes) 000844 1790921561.582662 read 0x012783 thru 0x019372 (0x6bf0 bytes) 000845 1790921561.583390 write 0x03a87e thru 0x03ffff (0x5782 bytes) 000846 1790921561.597995 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x8112120 (stale) Mirror 16: 0x8112120 (stale) Mirror 17: 0x8112120 (stale) Mirror 20: 0x8112120 (stale) Mirror 22: 0x8112120 (stale) Mirror 23: 0x8112120 (stale) Mirror 24: 0x8112120 (stale) Mirror 25: 0x8112120 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x8112120 (stale) Mirror 16: 0x8112120 (stale) Mirror 17: 0x8112120 (stale) Mirror 20: 0x8112120 (stale) Mirror 22: 0x8112120 (stale) Mirror 23: 0x8112120 (stale) Mirror 24: 0x8112120 (stale) Mirror 25: 0x8112120 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000848 1790921562.149670 punch from 0x0109d7 to 0x01b345 (0xa96f bytes) 000849 1790921562.155142 trunc from 0x040000 to 0x03e374 (0x1c8b bytes) 000850 1790921562.193049 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e375): Mirror 15: 0x9358a133 (stale) Mirror 16: 0x9358a133 (stale) Mirror 17: 0x9358a133 (stale) Mirror 22: 0x9358a133 (stale) Mirror 23: 0x9358a133 (stale) Mirror 24: 0x9358a133 (stale) Mirror 25: 0x9358a133 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e375): Mirror 15: 0x9358a133 (stale) Mirror 16: 0x9358a133 (stale) Mirror 17: 0x9358a133 (stale) Mirror 22: 0x9358a133 (stale) Mirror 23: 0x9358a133 (stale) Mirror 24: 0x9358a133 (stale) Mirror 25: 0x9358a133 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e375: skipped 000851 1790921562.830887 punch from 0x03b4ab to 0x03e374 (0x2eca bytes) 000852 1790921562.844800 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e375): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e375): Mirror 15: 0x9358a133 (stale) Mirror 16: 0x9358a133 (stale) Mirror 17: 0x9358a133 (stale) Mirror 22: 0x9358a133 (stale) Mirror 23: 0x9358a133 (stale) Mirror 24: 0x9358a133 (stale) Mirror 25: 0x9358a133 (stale) CRC-32 checksum value for chunk [0, 0x3e375): Mirror 1: 0xb271722e Mirror 26: 0xb271722e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e375: skipped 000854 1790921563.792826 write 0x01d735 thru 0x02d2a6 (0xfb72 bytes) 000855 1790921563.895652 write 0x023959 thru 0x02b7b6 (0x7e5e bytes) 000858 1790921563.898898 mapread 0x020361 thru 0x0290b2 (0x8d52 bytes) 000861 1790921563.905692 read 0x009a28 thru 0x00e005 (0x45de bytes) 000862 1790921563.914924 read 0x025f66 thru 0x03310f (0xd1aa bytes) 000864 1790921563.925289 read 0x028458 thru 0x02fdb8 (0x7961 bytes) 000865 1790921563.926794 read 0x015fce thru 0x01b9f4 (0x5a27 bytes) 000867 1790921563.928227 read 0x00d49d thru 0x018228 (0xad8c bytes) 000869 1790921563.930112 write 0x0259f8 thru 0x028bbf (0x31c8 bytes) 000872 1790921563.936960 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3e375): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e375): Mirror 15: 0x9358a133 (stale) Mirror 16: 0x9358a133 (stale) Mirror 17: 0x9358a133 (stale) Mirror 22: 0x9358a133 (stale) Mirror 23: 0x9358a133 (stale) Mirror 24: 0x9358a133 (stale) Mirror 25: 0x9358a133 (stale) Mirror 26: 0xb271722e (stale) CRC-32 checksum value for chunk [0, 0x3e375): Mirror 1: 0xcc130438 Mirror 27: 0xcc130438 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e375: skipped 000874 1790921564.924468 trunc from 0x03e375 to 0x03d30f (0x1065 bytes) 000878 1790921565.016027 write 0x017572 thru 0x0213d0 (0x9e5f bytes) 000879 1790921565.026704 write 0x02bc73 thru 0x034711 (0x8a9f bytes) 000880 1790921565.037759 punch from 0x00ec25 to 0x013f17 (0x52f3 bytes) 000881 1790921565.054952 mapread 0x01fb20 thru 0x0280fb (0x85dc bytes) 000883 1790921565.075059 punch from 0x0271c3 to 0x02f57e (0x83bc bytes) 000884 1790921565.080194 write 0x03e9bb thru 0x03ffff (0x1645 bytes) 000885 1790921565.083186 mapread 0x033bf2 thru 0x03ef97 (0xb3a6 bytes) 000886 1790921565.096728 mapread 0x004c04 thru 0x0137ef (0xebec bytes) 000888 1790921565.146529 mapwrite 0x025911 thru 0x0336ec (0xdddc bytes) 000889 1790921565.308916 read 0x01ceb0 thru 0x01d49b (0x5ec bytes) 000893 1790921565.318845 trunc from 0x040000 to 0x032cbc (0xd343 bytes) 000895 1790921565.349348 mapwrite 0x016c3d thru 0x01d35d (0x6721 bytes) 000896 1790921565.444729 mapwrite 0x030d98 thru 0x03105d (0x2c6 bytes) 000897 1790921565.547320 mapread 0x00fffd thru 0x016650 (0x6654 bytes) 000899 1790921565.555382 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x32cbd): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32cbd): Mirror 15: 0x45ba0941 (stale) Mirror 16: 0x45ba0941 (stale) Mirror 17: 0x45ba0941 (stale) Mirror 22: 0x45ba0941 (stale) Mirror 23: 0x45ba0941 (stale) Mirror 24: 0x45ba0941 (stale) Mirror 25: 0x45ba0941 (stale) Mirror 26: 0xdcdf888e (stale) Mirror 27: 0xd0d2d8d7 (stale) CRC-32 checksum value for chunk [0, 0x32cbd): Mirror 1: 0x1401a344 Mirror 28: 0x1401a344 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32cbd: skipped 000900 1790921566.499191 mapread 0x030e7f thru 0x032cbc (0x1e3e bytes) 000902 1790921566.546720 trunc from 0x032cbd to 0x027e80 (0xae3c bytes) 000905 1790921566.658168 punch from 0x01f66d to 0x027e80 (0x8814 bytes) 000906 1790921566.664561 write 0x0213d3 thru 0x022e2a (0x1a58 bytes) 000907 1790921566.677568 trunc from 0x027e81 to 0x020da3 (0x70dd bytes) 000908 1790921566.701045 MIRROR_RESYNC 11 mirrors 000909 1790921568.276058 mapread 0x00d37f thru 0x0169f3 (0x9675 bytes) 000911 1790921568.310165 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) 000912 1790921568.906555 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) 000913 1790921569.447320 mapread 0x01ba2d thru 0x020da3 (0x5377 bytes) 000916 1790921569.478856 read 0x01c39f thru 0x020598 (0x41fa bytes) 000917 1790921569.479778 write 0x0252c6 thru 0x029b95 (0x48d0 bytes) 000918 1790921569.592324 punch from 0x019956 to 0x01cad3 (0x317e bytes) 000919 1790921569.643232 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29b96): Mirror 15: 0x983197b4 (stale) Mirror 17: 0x983197b4 (stale) Mirror 22: 0x983197b4 (stale) Mirror 23: 0x983197b4 (stale) Mirror 24: 0x983197b4 (stale) Mirror 27: 0x983197b4 (stale) Mirror 28: 0x983197b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b96): Mirror 15: 0x983197b4 (stale) Mirror 17: 0x983197b4 (stale) Mirror 22: 0x983197b4 (stale) Mirror 23: 0x983197b4 (stale) Mirror 24: 0x983197b4 (stale) Mirror 27: 0x983197b4 (stale) Mirror 28: 0x983197b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b96: skipped 000921 1790921570.871014 write 0x007273 thru 0x00b85b (0x45e9 bytes) 000923 1790921570.883134 mapread 0x00268e thru 0x004738 (0x20ab bytes) 000925 1790921570.919094 read 0x004b7e thru 0x0099e1 (0x4e64 bytes) 000926 1790921570.919841 trunc from 0x029b96 to 0x0040f6 (0x25a9f bytes) 000928 1790921570.963369 mapwrite 0x019410 thru 0x0264c3 (0xd0b4 bytes) 000929 1790921571.173864 MIRROR_RESYNC 8 mirrors 000930 1790921572.752003 trunc from 0x0264c4 to 0x01f535 (0x6f8e bytes) 000931 1790921572.814209 read 0x01d39b thru 0x01e974 (0x15da bytes) 000932 1790921572.818219 mapwrite 0x036be3 thru 0x03e278 (0x7696 bytes) 000933 1790921572.915229 punch from 0x03b4c1 to 0x03e278 (0x2db8 bytes) 000934 1790921572.922109 read 0x02ea81 thru 0x033b0e (0x508e bytes) 000935 1790921572.939230 punch from 0x03382f to 0x03e278 (0xaa4a bytes) 000936 1790921572.947933 write 0x035eaa thru 0x03e739 (0x8890 bytes) 000937 1790921572.958012 write 0x009b17 thru 0x00fa1d (0x5f07 bytes) 000938 1790921572.959703 trunc from 0x03e73a to 0x000c75 (0x3dac4 bytes) 000940 1790921572.982871 mapread 0x00054c thru 0x000c75 (0x72a bytes) 000941 1790921572.989676 MIRROR_RESYNC 8 mirrors 000943 1790921574.486148 punch from 0x000625 to 0x000c75 (0x651 bytes) 000944 1790921574.667163 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc76): Mirror 17: 0x931914ea (stale) Mirror 22: 0x931914ea (stale) Mirror 23: 0x931914ea (stale) Mirror 24: 0x931914ea (stale) Mirror 27: 0x931914ea (stale) Mirror 28: 0x931914ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc76): Mirror 17: 0x931914ea (stale) Mirror 22: 0x931914ea (stale) Mirror 23: 0x931914ea (stale) Mirror 24: 0x931914ea (stale) Mirror 27: 0x931914ea (stale) Mirror 28: 0x931914ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc76: skipped 000947 1790921575.065091 mapread 0x000ab2 thru 0x000c75 (0x1c4 bytes) 000948 1790921575.070932 write 0x011f66 thru 0x019eb5 (0x7f50 bytes) 000949 1790921575.072913 mapwrite 0x03bd37 thru 0x03e721 (0x29eb bytes) 000951 1790921575.189854 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e722): Mirror 17: 0x30e8a4b (stale) Mirror 22: 0x30e8a4b (stale) Mirror 23: 0x30e8a4b (stale) Mirror 24: 0x30e8a4b (stale) Mirror 27: 0x30e8a4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e722): Mirror 17: 0x30e8a4b (stale) Mirror 22: 0x30e8a4b (stale) Mirror 23: 0x30e8a4b (stale) Mirror 24: 0x30e8a4b (stale) Mirror 27: 0x30e8a4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e722: skipped 000953 1790921575.624874 write 0x015342 thru 0x020d75 (0xba34 bytes) 000954 1790921575.636325 write 0x02bf5e thru 0x033cef (0x7d92 bytes) 000957 1790921575.648051 mapread 0x01e394 thru 0x024f9a (0x6c07 bytes) 000959 1790921575.671184 write 0x024f7d thru 0x0296fa (0x477e bytes) 000960 1790921575.673160 trunc from 0x03e722 to 0x03cf63 (0x17be bytes) 000963 1790921575.689974 mapwrite 0x00668c thru 0x00fc01 (0x9576 bytes) 000964 1790921575.749055 mapwrite 0x034173 thru 0x03d771 (0x95ff bytes) 000965 1790921575.821987 trunc from 0x03d772 to 0x01b9a9 (0x21dc8 bytes) 000966 1790921575.839128 write 0x0317b9 thru 0x035ad0 (0x4318 bytes) 000967 1790921575.840435 trunc from 0x035ad1 to 0x02f581 (0x654f bytes) 000968 1790921575.858156 mapwrite 0x0228f6 thru 0x02b955 (0x9060 bytes) 000969 1790921575.936769 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f582): Mirror 22: 0xb7fc6bcf (stale) Mirror 23: 0xb7fc6bcf (stale) Mirror 24: 0xb7fc6bcf (stale) Mirror 27: 0xb7fc6bcf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f582): Mirror 22: 0xb7fc6bcf (stale) Mirror 23: 0xb7fc6bcf (stale) Mirror 24: 0xb7fc6bcf (stale) Mirror 27: 0xb7fc6bcf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f582: skipped 000970 1790921576.338098 trunc from 0x02f582 to 0x0113a1 (0x1e1e0 bytes) 000971 1790921576.358834 mapwrite 0x02d287 thru 0x0302bd (0x3037 bytes) 000972 1790921576.471790 mapwrite 0x0169bf thru 0x01d9df (0x7021 bytes) 000974 1790921576.737446 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x302be): Mirror 22: 0xce7544cc (stale) Mirror 24: 0xce7544cc (stale) Mirror 27: 0xce7544cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302be): Mirror 22: 0xce7544cc (stale) Mirror 24: 0xce7544cc (stale) Mirror 27: 0xce7544cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302be: skipped 000977 1790921577.461238 mapread 0x0005a6 thru 0x003156 (0x2bb1 bytes) 000979 1790921577.524910 mapread 0x00c404 thru 0x0136fb (0x72f8 bytes) 000980 1790921577.575944 punch from 0x02809c to 0x02ce8c (0x4df1 bytes) 000982 1790921577.584244 mapwrite 0x00023a thru 0x00b6a5 (0xb46c bytes) 000983 1790921577.724124 punch from 0x00c353 to 0x0191d0 (0xce7e bytes) 000985 1790921577.729754 mapwrite 0x02b2e0 thru 0x02ca40 (0x1761 bytes) 000986 1790921577.893595 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x302be): Mirror 22: 0xce7544cc (stale) Mirror 24: 0xce7544cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302be): Mirror 22: 0xce7544cc (stale) Mirror 24: 0xce7544cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302be: skipped 000987 1790921578.407925 punch from 0x01b2f9 to 0x023bfc (0x8904 bytes) 000988 1790921578.416555 mapread 0x00a8ac thru 0x01a0ed (0xf842 bytes) 000991 1790921578.495608 mapread 0x01b048 thru 0x028f9e (0xdf57 bytes) 000992 1790921578.536654 read 0x008db5 thru 0x00bd74 (0x2fc0 bytes) 000993 1790921578.537304 mapwrite 0x0149bd thru 0x0150e0 (0x724 bytes) 000995 1790921578.621231 mapread 0x0090d5 thru 0x00f0ac (0x5fd8 bytes) 000996 1790921578.627581 trunc from 0x0302be to 0x021e38 (0xe485 bytes) 000997 1790921578.663994 mapread 0x0058ad thru 0x012dc0 (0xd514 bytes) 001000 1790921578.667853 mapwrite 0x012e03 thru 0x015d3a (0x2f38 bytes) All operations completed A-OK! PASS 70a (140s) == sanity-flr test 71: check mirror extend parent fid ==== 02:13:06 (1790921586) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 02:13:11 (1790921591) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.497013 s, 4.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293275 s, 3.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 02:13:23 (1790921603) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7388192 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8129376 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8161856 .. Extending file size to 8606976 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8655360 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8673152 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8742720 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58227 58228 58229 58233 58234 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) 192.168.202.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (90s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:14:53 (1790921693) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2614240 .. Extending file size to 1308640 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5165152 .. Extending file size to 9146816 .. Extending file size to 7572768 .. Extending file size to 8033504 .. Extending file size to 8126848 .. Extending file size to 8375808 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8913344 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 59664 59665 59666 59669 59670 done 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) 192.168.202.109@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (88s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:16:21 (1790921781) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118299 s, 17.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.311356 s, 6.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xb2797d06 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:16:43 (1790921803) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0768071 s, 27.3 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 02:16:51 (1790921811) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x255:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:16:58 (1790921818) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202238 s, 10.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:17:07 (1790921827) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.394617 s, 5.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x311:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:17:18 (1790921838) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.328075 s, 6.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x312:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:17:54 (1790921874) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181904 s, 5.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x313:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:18:59 (1790921939) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.319037 s, 6.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x315:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:19:59 (1790921999) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. Creating new pool test_204f oleg209-server: Pool lustre.test_204f created Adding targets to pool oleg209-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg209-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175517 s, 11.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg209-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg209-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg209-server: Pool lustre.test_204f destroyed PASS 204f (88s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:21:27 (1790922087) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:21:35 (1790922095) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:21:43 (1790922103) oleg209-server: Pool lustre.test_206 created Adding targets to pool oleg209-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg209-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg209-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg209-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg209-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:22:08 (1790922128) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412234 s, 25.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203293 s, 51.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb5f57fbd Mirror 3: 0xb5f57fbd PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:22:21 (1790922141) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:22:26 (1790922146) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:22:30 (1790922150) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.248374 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1790000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1790000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1790000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1df0000 Read this (4 bytes): BBBB 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:23:10 (1790922190) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0377117 s, 217 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:23:18 (1790922198) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083266 s, 12.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x27a lmm_fid: [0x200000404:0x27a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x27a lmm_fid: [0x200000404:0x27a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x321:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834554 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 02:23:20.000000000 -0400 Modify: 2026-10-02 02:23:20.000000000 -0400 Change: 2026-10-02 02:23:22.000000000 -0400 Birth: 2026-10-02 02:23:21.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:23:28 (1790922208) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:23:32 (1790922212) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.45482 s, 90.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0525525 s, 77.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 2 02:23 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x27c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 2 02:23 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:23:40 (1790922220) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:23:47 (1790922227) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 2007 sec =========== 02:23:56 (1790922236) === sanity-flr: start cleanup 02:23:57 (1790922237) === === sanity-flr: finish cleanup 02:24:03 (1790922243) ===